From 5535430179a575517d8c1ed5e1e15bc75d1cbe1c Mon Sep 17 00:00:00 2001 From: Chayapat Sriwattanachote Date: Wed, 18 Nov 2026 10:29:16 +0700 Subject: [PATCH 01/11] docs(voice): select bounded protected-host local speech contract --- ...16-bounded-local-dictation-and-playback.md | 57 +++++++++++++++++++ docs/contracts/s16-voice-v1.md | 46 +++++++++++++++ 2 files changed, 103 insertions(+) create mode 100644 docs/adr/0096-s16-bounded-local-dictation-and-playback.md create mode 100644 docs/contracts/s16-voice-v1.md diff --git a/docs/adr/0096-s16-bounded-local-dictation-and-playback.md b/docs/adr/0096-s16-bounded-local-dictation-and-playback.md new file mode 100644 index 00000000..e8267053 --- /dev/null +++ b/docs/adr/0096-s16-bounded-local-dictation-and-playback.md @@ -0,0 +1,57 @@ +--- +status: selected engineering contract for implementation; no voice availability claimed yet +date: 2026-11-18 +scope: VOICE16-01 (#422); S16-AC15; FR-06/11/13/14/19/24/25/37/38; NFR-05/06/07/09/11 +--- + +# Bounded local dictation and reply playback + +## Required outcome and current baseline + +S16-AC15 and the authored [voice experience](../design/S16-rakazo-experience.md#voice-and-clients) require intentional microphone capture, visible recording/transcription, an editable transcript, cancellable playback and finite usage. Text remains usable. Audio/transcribed approval words cannot invoke an exact protected decision. Continuous calls belong to #423; native account/device evidence belongs to #431. The primary agent owns implementation/design under the current AGENTS agreement. + +At main `be21d747d25f4a6a234f057d8b35362501b03a7f`, the shared client (#395) and connections (#417) prerequisites are merged/closed. The React composer, guarded owner projections, command identities and exact decision cards are available. There is no audio implementation; `tests/chat-voice*` tests prose style. #422 was unassigned with no competing branch/PR; it is now owned on an issue-specific worktree. Existing worktrees are untouched. + +## Alternatives and source observations + +| Choice | Benefits and limits | Selection | +| --- | --- | --- | +| Browser/system speech recognition and synthesis | Little server infrastructure. Recognition may use a browser/vendor service or lack a language/platform; synthesis voices can also be remote. A system API is not evidence of local custody | Do not silently use it as local speech. A future explicit device-local adapter must prove its locality and availability | +| Protected provider STT/TTS | Broad language/voice coverage; requires exact account/recipient, billing scope, current private-source checks and uncertain-usage handling | Keep as a future adapter option, not an automatic fallback or a prerequisite for configured local voice | +| Local CLI STT/TTS on the protected Jit host | Actual speech without a new external recipient or paid account. Operator supplies verified binaries/model; resource limits and lifecycle are Jit's responsibility. Recognition/voice quality varies | Select `whisper.cpp` transcription and `espeak-ng` synthesis on Linux, with an explicit `say` synthesis adapter for macOS development/evidence | + +The selected local route satisfies #422's configured-local-or-provider choice. It does not claim every provider or device is supported. The host uses fixed argument arrays, private bounded files and finite processes, never a model-provided command or a working-VM execution backend. Schway continues to own model/tool execution; this owner-client audio adapter introduces no provider loop or private Schway import. + +Rakazo inspection: repository `elie222/rakazo`, immutable commit `abf3c08145a05cba92a912d6c24fdfb5f569d0e0`, tree `f8ca5adc96366b722bb60186c21cb29cec0f5390` (complete tree, not truncated). Read without executing upstream scripts: + +| Source | SHA-256 | Observation | +| --- | --- | --- | +| `packages/adapters/src/openai-compatible-voice.ts` | `33c9fd666df55f36305b26e1a0f9b6deaf0e73e36e07e0382c64f51d5f154c76` | Explicit compatible server/model credentials, bounded transcription/synthesis calls; not Jit's recipient/budget authority | +| `packages/adapters/src/voice-http.ts` | `ceda3d36f626a5893e808f8e94ceb2edd7ff853a234c718165697843359fb80e` | Finite request deadlines and response byte limits are useful concepts | +| `apps/desktop/native/speech/main.swift` | `099d8862dae69ebbd077b352c30afcbf21e949c7008cba1c08c56f0225295b30` | Requests on-device recognition when supported, otherwise permits system service; that fallback cannot establish Jit's local-only route | + +These are comparison observations, not upstream vulnerability claims. No upstream source/assets/dependencies are copied. Existing Apache-2.0 inventory/notices remain unchanged; future copied files follow ADR 0074/0080. CLI binaries, voice data and model weights are separately operator-supplied artifacts; each installation must retain their applicable licenses/provenance rather than infer them from Jit's MIT license. + +## Selected ownership and custody + +The proposed [s16-voice/v1](../contracts/s16-voice-v1.md) owns finite client capture sessions and local speech usage. It adds no Bot tool, message admission or approval authority. The configured voice route is installation-owned, opt-in and contains no credentials. Its artifact paths/hashes are protected configuration; clients get labels, capabilities and limits only. No automatic binary/model download or installation activation occurs. + +The owner starts dictation with a deliberate Mic press. The authenticated service reserves finite local quota before capture. The browser then requests microphone permission, captures bounded PCM and stops every track/audio node on Stop, Cancel, permission loss, page/target/session change or deadline. Cancelled/late permission promises immediately stop any subsequently supplied stream. No browser speech-recognition service is contacted. + +Stop submits a bounded canonical WAV to the same authenticated Jit service for one fixed local transcription. A transcript is untrusted draft data. It is not a conversation message until the owner uses ordinary Send; editing and cancellation create no Bot work. Spoken/quoted "approve" cannot call `decide`, `decideAction` or change purpose. Existing file-bearing/steering semantics remain explicit rather than being inferred from speech. + +Playback selects the exact currently guarded reply by conversation/request/revision/text digest; it does not accept arbitrary unguarded text from the page. Recheck owner/session and source publication before local synthesis and before publishing audio. Interrupt stops playback immediately and fences pending playback responses; underlying Bot work keeps its existing state. Local synthesis is not evidence that an effect succeeded. + +## Bounds, uncertainty and compatibility + +Use finite per-capture duration, WAV bytes, transcript/reply bytes, local process time, synthesized audio duration/bytes and persistent per-day local speech quota. Configuration can narrow published hard limits. Unknown/failed process usage retains its conservative reservation; do not reset quota on restart or confuse audio seconds with measured CPU/GPU time, provider tokens or a zero monetary bill. Local route loss/budget exhaustion ends voice with a safe code while ordinary text remains usable. No fallback creates a new account/recipient or raises limits. + +Audio/transcript scratch is private and short-lived, never in model prompts, working VM, exports, raw errors or logs. Submitted text uses existing conversation retention. Metadata-only quota receipts remain protected; abandoned pending operations are terminal after restart and cannot silently replay. Shutdown kills and actually joins local children before removing scratch/releasing capacity. Browser stop and server settlement are separate facts. + +This is a new optional voice journal/port, not a migration of CHAT, TEAM, MEM, FILE or policy meaning. S14 stays default; S15 opt-in and omitted-runtime compatibility stay. No existing owner host/account/policy is changed by implementation. Packaging verifies the optional feature without bundling a model/binary or asserting device acceptance. #422 closes only after actual composed client/service and native speech evidence plus required checks/review; this document alone does not complete it. + +## Verification obligations + +Use deterministic PCM/command fixtures, fake clocks and barriers for quota, cancellation, lost responses, stale generations and current owner/publication checks. Browser tests exercise the real bundle in English/Thai and mobile/keyboard layouts, with fake media devices rather than a real microphone. Native opt-in evidence must transcribe actual known speech and synthesize an actual guarded reply through configured local executables, recording exact binary/model/build hashes. Label recognition accuracy, device permissions and all untested platforms honestly. #423 retains continuous-call/barge-in/reconnect behavior. + +Protocol references inspected 2026-11-18: [Media Capture and Streams](https://www.w3.org/TR/mediacapture-streams/), [whisper.cpp CLI implementation](https://github.com/ggml-org/whisper.cpp/blob/master/examples/cli/cli.cpp), [espeak-ng command reference](https://github.com/espeak-ng/espeak-ng/blob/master/src/espeak-ng.1.ronn). Implementations must validate their exact supplied build; these moving references are not installation pins. diff --git a/docs/contracts/s16-voice-v1.md b/docs/contracts/s16-voice-v1.md new file mode 100644 index 00000000..d8e1c4ee --- /dev/null +++ b/docs/contracts/s16-voice-v1.md @@ -0,0 +1,46 @@ +# s16-voice/v1 + +Status: selected contract for #422 implementation ([ADR 0096](../adr/0096-s16-bounded-local-dictation-and-playback.md)); **not an implemented capability yet**. #423 owns continuous calls. This draft records decisions before behavior changes and must be reconciled with actual exported signatures/endpoints/tests before delivery. + +## Protected host route + +Compose an optional local voice service in S15 and add an optional `voice` capability to the existing shared client. Older services/clients keep text behavior. No configured route means voice unavailable, never an implicit browser/cloud service. A route names pinned executable/model artifacts, STT language, synthesis kind/voice and finite local quota. Protected paths and raw CLI output never enter capability JSON. Configuration is explicit host-owned state, not a model/tool payload. + +Linux route: `whisper.cpp` CLI with fixed model/file/output/language/no-GPU/thread arguments, plus `espeak-ng` file-to-WAV synthesis. macOS evidence may explicitly select `say` file-to-WAV synthesis; this does not claim Linux availability. No shell parsing, arbitrary flags, URL input, inherited provider credentials or automatic model download. Verify exact configured artifact hashes before use. Operator-supplied libraries/model/voice data remain a trusted installation dependency; do not claim the CLI wrapper is a sandbox against a malicious executable. + +## Owner-client operations + +All operations use the current shared-client owner session, exact Host/Origin, CSRF for POST, no cross-site access, no-store responses, finite streamed bodies and reauthentication after asynchronous boundaries. No voice endpoint accepts an approval command or serialized actor. + +| Operation | Binding and result | +| --- | --- | +| Read capability/usage | Actual configured route health, finite limits, local audio-second quota and pending/unknown reservations. Read starts no capture/process | +| Begin dictation | One exact command ID, conversation and current session generation; reserve bounded local capture quota. Returns opaque capture identity, expiry and permitted duration/bytes; starts no Bot work | +| Transcribe | One capture identity and strict PCM WAV bytes; identity/hash conflict refuses. Actual owner/route/capacity checks precede process start and transcript publication. Returns editable untrusted draft text plus observed/unknown usage | +| Cancel | Fence current capture/transcription, stop and join actual local process, discard scratch/draft. A late result cannot revive it. Does not cancel a Bot request | +| Synthesize reply | One identity with exact conversation/request/revision/text hash. Reacquire current guarded owner projection; synthesize only its published result. Return bounded WAV after current authorization/source checks | +| Interrupt playback | Stop client audio immediately, revoke its object URL and fence pending synthesis/play responses; underlying work remains separate | + +These are semantic operations; final HTTP paths and exported types are frozen with the implementation. Client retries never invent an identity or automatically replay speech. No transcript is automatically submitted to CHAT. Owner Send uses the ordinary frozen message command/receipt and unchanged exact-decision surface. Cancel before Send creates no conversation request/work. + +## Audio and finite usage + +Capture uses explicit `getUserMedia` after a deliberate Mic action, with visible Recording/elapsed time/Stop/Cancel. A bounded PCM conversion emits canonical RIFF/WAVE, PCM format 1, mono, 16-bit little-endian at 16,000 Hz. Validate RIFF length, exact format/data layout, alignment, sample count and hard duration/byte limits before CLI input. Reject malformed/compressed/oversized or trailing data rather than trusting MIME or client duration. No private audio decoding library or arbitrary attachment parser runs for capture. + +Hard per-capture ceiling: 30,000 ms / 960,044 canonical WAV bytes. Transcript is at most 16 KiB UTF-8 and still subject to ordinary message limits. Synthesis accepts at most 2,000 characters / 8 KiB UTF-8 of one current result, with at most 60 seconds / 4 MiB output and a finite process deadline. Longer replies remain readable; do not silently label truncated speech as a complete result. Final implementation may narrow bounds and must publish the actual limits before capture/playback. + +Configured quota is local audio seconds per UTC day, persisted outside the working VM. Reserve worst-case capacity before capture/synthesis; update observed accepted/generated audio seconds only after real evidence. Unknown processing keeps its conservative charge/reservation, including restart. Local audio seconds are not measured compute time, provider billing or tokens. A refused/not-started operation can release unused capacity; cancellation after actual processing is not proof of zero usage. Quota exhaustion or route failure stops voice without disabling text or switching provider. + +Only one owner capture/playback lifecycle is active per client generation. Permission denial/revocation, track `ended`/unusable capture, cancellation, limit, target change, logout and unmount stop all tracks/nodes and clear buffers. An unresolved permission prompt can outlive a cancellation: any later stream must be stopped without adoption. Stop does not rely on a `MediaStreamTrack` ended event firing for application-initiated stop. Browser resources and pending responses are fenced independently of server process settlement. + +## Custody, publication and recovery + +Raw audio stays in client memory and protected service scratch. The local engine receives only bounded private files and a minimal environment; raw output is never returned as an error. Scratch is removed after actual process settlement; startup reaps abandoned scratch without following arbitrary paths. No audio/transcript is placed in the working VM, native model history, logs or ordinary exports. A submitted transcript becomes ordinary owner text under existing CHAT custody. Browser/device-local memory cannot be remotely recalled after disclosure. + +The service owns a separate versioned voice/quota journal with bounded metadata and no audio. It retains original command identities and conservative charges across process loss. A restarted pending local operation becomes terminal/unknown, with no reconstructed execution. Missing/expired draft/audio results are reported honestly; automatic re-recording, re-synthesis or re-submission is forbidden. #423 will extend continuity for confirmed continuous-call turns using the existing durable CHAT receipt, rather than treating a voice response as work acceptance. + +Private reply publication is always reacquired through the owning conversation service, including before returning generated audio. Source withdrawal, owner/session revocation, changed result/revision, changed route artifact or generation prevents stale publication. Interruption invalidates client playback without claiming remote work or effects stopped. Audio content, transcripts, quoted commands and model output cannot approve, change scope or configure a route. + +## Delivery evidence + +Tests must exercise the actual configured local adapter/service and ordinary Send/decision ports, not only a reducer or a mock successful transcript. Deterministic fixtures cover strict WAV parsing, finite quotas across restart, duplicate/conflicting commands, late permission/result rejection, denied/revoked capture, exact reply/current-source checks, provider/process loss and no work before Send. Real English/Thai bundle tests cover keyboard/mobile/error states, interrupted playback, text fallback and transcribed approval words. Opt-in local speech evidence records actual known speech transcription and reply audio with exact executable/model hashes. Physical native-device/permission acceptance remains #431; source presence or this contract is not completion. From ee9f0da687d8b3c39a8fe086f047d3b43b2849eb Mon Sep 17 00:00:00 2001 From: Chayapat Sriwattanachote Date: Wed, 18 Nov 2026 10:30:26 +0700 Subject: [PATCH 02/11] feat(voice): bound canonical PCM capture bytes before local transcription --- src/s16/voice/wav.ts | 43 +++++++++++++++++++++++++++++++++++ tests/s16-voice-audio.test.ts | 38 +++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+) create mode 100644 src/s16/voice/wav.ts create mode 100644 tests/s16-voice-audio.test.ts diff --git a/src/s16/voice/wav.ts b/src/s16/voice/wav.ts new file mode 100644 index 00000000..59e11e08 --- /dev/null +++ b/src/s16/voice/wav.ts @@ -0,0 +1,43 @@ +/** Canonical capture format: mono PCM16 at 16 kHz, bounded independently of client MIME/duration. */ +export const captureAudio = Object.freeze({ sampleRate: 16000, channels: 1, bits: 16, durationMs: 30000, bytes: 960044 }); + +/** Safe public failure code; never carries audio, transcript or private host paths. */ +export class VoiceAudioError extends Error { + constructor(readonly code: 'voice_audio_invalid' | 'voice_audio_limit') { super(code); } +} + +const tag = (bytes: Uint8Array, offset: number, value: string): boolean => [...value].every((letter, index) => bytes[offset + index] === letter.charCodeAt(0)); +const invalid = (): never => { throw new VoiceAudioError('voice_audio_invalid'); }; +const limit = (): never => { throw new VoiceAudioError('voice_audio_limit'); }; + +/** + * Validate the complete canonical WAV before a local engine sees bytes. Rejects other codecs/chunks, + * false lengths, alignment, trailing data and excess samples. Metadata is derived from actual bytes; + * the caller still owns snapshotting its input before an async boundary. + */ +export function captureWavInfo(bytes: Uint8Array, maxDurationMs: number = captureAudio.durationMs): { frames: number; durationMs: number; bytes: number } { + if (!Number.isSafeInteger(maxDurationMs) || maxDurationMs < 1 || maxDurationMs > captureAudio.durationMs) invalid(); + if (bytes.byteLength > captureAudio.bytes || bytes.byteLength > 44 + Math.floor(maxDurationMs * captureAudio.sampleRate / 1000) * 2) limit(); + if (bytes.byteLength < 46 || !tag(bytes, 0, 'RIFF') || !tag(bytes, 8, 'WAVE') || !tag(bytes, 12, 'fmt ') || !tag(bytes, 36, 'data')) invalid(); + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength), size = bytes.byteLength - 44; + if (view.getUint32(4, true) !== bytes.byteLength - 8 || view.getUint32(16, true) !== 16 || view.getUint16(20, true) !== 1 + || view.getUint16(22, true) !== 1 || view.getUint32(24, true) !== captureAudio.sampleRate || view.getUint32(28, true) !== 32000 + || view.getUint16(32, true) !== 2 || view.getUint16(34, true) !== 16 || view.getUint32(40, true) !== size || size % 2 !== 0) invalid(); + return { frames: size / 2, durationMs: size / 2 * 1000 / captureAudio.sampleRate, bytes: bytes.byteLength }; +} + +/** Encode already resampled mono 16 kHz capture. Finite samples are clipped to PCM16; no codec/dependency is used. */ +export function encodeCaptureWav(samples: Float32Array): Uint8Array { + if (samples.length > captureAudio.sampleRate * captureAudio.durationMs / 1000) limit(); + if (!samples.length) invalid(); + const bytes = new Uint8Array(44 + samples.length * 2), view = new DataView(bytes.buffer); + const write = (offset: number, text: string) => { for (let index = 0; index < text.length; index++) bytes[offset + index] = text.charCodeAt(index); }; + write(0, 'RIFF'); view.setUint32(4, bytes.length - 8, true); write(8, 'WAVE'); write(12, 'fmt '); + view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, 1, true); view.setUint32(24, captureAudio.sampleRate, true); + view.setUint32(28, 32000, true); view.setUint16(32, 2, true); view.setUint16(34, 16, true); write(36, 'data'); view.setUint32(40, samples.length * 2, true); + for (let index = 0; index < samples.length; index++) { + const sample = samples[index]!; if (!Number.isFinite(sample)) invalid(); + const clipped = Math.min(1, Math.max(-1, sample)); view.setInt16(44 + index * 2, Math.round(clipped * (clipped < 0 ? 32768 : 32767)), true); + } + return bytes; +} diff --git a/tests/s16-voice-audio.test.ts b/tests/s16-voice-audio.test.ts new file mode 100644 index 00000000..da880e6d --- /dev/null +++ b/tests/s16-voice-audio.test.ts @@ -0,0 +1,38 @@ +import { expect, test } from 'bun:test'; +import { captureAudio, captureWavInfo, encodeCaptureWav } from '../src/s16/voice/wav'; + +// Independently specified PCM fixture: [-32768, 0, 32767] in a standard 16 kHz mono WAV. +const fixture = (): Uint8Array => Uint8Array.from(Buffer.from('524946462a00000057415645666d74201000000001000100803e0000007d000002001000646174610600000000800000ff7f', 'hex')); + +test('actual header/data bytes determine capture duration, including nonzero byte-offset views', () => { + const wav = fixture(), wrapped = new Uint8Array(wav.length + 14); wrapped.set(wav, 7); + expect(captureWavInfo(wav)).toEqual({ frames: 3, durationMs: 0.1875, bytes: 50 }); + expect(captureWavInfo(wrapped.subarray(7, 7 + wav.length))).toEqual(captureWavInfo(wav)); + expect(encodeCaptureWav(new Float32Array([-1, 0, 1]))).toEqual(wav); +}); + +test('codec, layout, declared sizes and trailing bytes cannot smuggle another input into the local engine', () => { + for (const [offset, value] of [[0, 0], [8, 0], [12, 0], [16, 18], [20, 3], [22, 2], [24, 0], [28, 1], [32, 4], [34, 8], [36, 0], [40, 4], [4, 0]] as const) { + const bytes = fixture(); bytes[offset] = value; expect(() => captureWavInfo(bytes)).toThrow('voice_audio_invalid'); + } + expect(() => captureWavInfo(new Uint8Array())).toThrow('voice_audio_invalid'); + expect(() => captureWavInfo(fixture().subarray(0, 49))).toThrow('voice_audio_invalid'); + const trailing = new Uint8Array(52); trailing.set(fixture()); expect(() => captureWavInfo(trailing)).toThrow('voice_audio_invalid'); +}); + +test('exact hard duration/byte bound accepts 30 seconds and refuses an additional sample', () => { + const maximum = encodeCaptureWav(new Float32Array(480000)); + expect(maximum.length).toBe(captureAudio.bytes); expect(captureWavInfo(maximum).durationMs).toBe(30000); + expect(() => encodeCaptureWav(new Float32Array(480001))).toThrow('voice_audio_limit'); + const tooLarge = new Uint8Array(maximum.length + 2); tooLarge.set(maximum); + expect(() => captureWavInfo(tooLarge)).toThrow('voice_audio_limit'); + expect(() => captureWavInfo(encodeCaptureWav(new Float32Array(16001)), 1000)).toThrow('voice_audio_limit'); + expect(captureWavInfo(encodeCaptureWav(new Float32Array(16000)), 1000).durationMs).toBe(1000); +}); + +test('invalid limits/nonfinite audio refuse, while finite out-of-range device samples clip without wrapping', () => { + for (const duration of [0, -1, 0.5, NaN, Infinity, 30001]) expect(() => captureWavInfo(fixture(), duration)).toThrow('voice_audio_invalid'); + for (const value of [NaN, Infinity, -Infinity]) expect(() => encodeCaptureWav(new Float32Array([value]))).toThrow('voice_audio_invalid'); + expect(() => encodeCaptureWav(new Float32Array())).toThrow('voice_audio_invalid'); + expect(encodeCaptureWav(new Float32Array([-4, 0, 4]))).toEqual(fixture()); +}); From bf304977e6de024b7e26874d08e92b66aa942642 Mon Sep 17 00:00:00 2001 From: Chayapat Sriwattanachote Date: Wed, 18 Nov 2026 10:53:59 +0700 Subject: [PATCH 03/11] feat(voice): supervise pinned local speech processes through cancellation and controller loss --- src/s16/voice/local-worker.ts | 46 ++++++++++ src/s16/voice/local.ts | 138 +++++++++++++++++++++++++++++ tests/fixtures/s16-voice-parent.ts | 9 ++ tests/s16-voice-local.test.ts | 106 ++++++++++++++++++++++ 4 files changed, 299 insertions(+) create mode 100644 src/s16/voice/local-worker.ts create mode 100644 src/s16/voice/local.ts create mode 100644 tests/fixtures/s16-voice-parent.ts create mode 100644 tests/s16-voice-local.test.ts diff --git a/src/s16/voice/local-worker.ts b/src/s16/voice/local-worker.ts new file mode 100644 index 00000000..58bf62ea --- /dev/null +++ b/src/s16/voice/local-worker.ts @@ -0,0 +1,46 @@ +// Private finite process supervisor. Only its parent supplies a job over a fresh anonymous pipe. +// Keeping stdin open is the parent's liveness lease: EOF/SIGTERM/deadline stops and joins the CLI. +import { lstatSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; + +if (import.meta.main) { + if (process.argv[2] !== '--local-voice-worker') process.exit(2); + let child: ReturnType | undefined, reason: string | null = null, finished = false, directory: string | undefined; + const stop = (code: string) => { reason ??= code; if (child && !finished) { try { child.kill('SIGKILL'); } catch { /* Exit may have won the race. */ } } }; + process.on('SIGTERM', () => stop('cancelled')); process.on('SIGINT', () => stop('cancelled')); + const respond = (state: string) => { + finished = true; + // A restarted service may remove this private scratch only after the actual CLI was joined. + if (directory) writeFileSync(join(directory, 'settled.json'), JSON.stringify({ state, worker: process.pid, child: child?.pid ?? null }), { mode: 0o600 }); + process.stdout.write(JSON.stringify({ state }) + '\n'); process.exit(0); + }; + const run = async (raw: string) => { + let deadline: ReturnType | undefined, size: ReturnType | undefined; + try { + const job = JSON.parse(raw) as { argv: string[]; cwd: string; timeoutMs: number; output: string; bytes: number }; + if (!job || Object.keys(job).sort().join() !== 'argv,bytes,cwd,output,timeoutMs' || !Array.isArray(job.argv) || !job.argv.length || job.argv.length > 32 + || job.argv.some(value => typeof value !== 'string' || !value || value.length > 4096 || value.includes('\0')) || !job.argv[0]!.startsWith('/') + || typeof job.cwd !== 'string' || !job.cwd.startsWith('/') || typeof job.output !== 'string' || !job.output.startsWith(job.cwd + '/') + || !Number.isSafeInteger(job.timeoutMs) || job.timeoutMs < 1 || job.timeoutMs > 120000 || !Number.isSafeInteger(job.bytes) || job.bytes < 1 || job.bytes > 4 * 1024 * 1024) return respond('failed'); + directory = job.cwd; + if (reason) return respond(reason); + child = Bun.spawn(job.argv, { cwd: job.cwd, stdin: 'ignore', stdout: 'ignore', stderr: 'ignore', env: { PATH: '/usr/bin:/bin', LANG: 'C.UTF-8', LC_ALL: 'C.UTF-8', TMPDIR: job.cwd } }); + deadline = setTimeout(() => stop('timeout'), job.timeoutMs); + size = setInterval(() => { try { if (lstatSync(job.output).size > job.bytes) stop('limit'); } catch { /* The CLI may not have created its output yet. */ } }, 20); + const code = await child.exited; + respond(reason ?? (code === 0 ? 'completed' : 'failed')); + } catch { stop('failed'); if (child) await child.exited; respond(reason ?? 'failed'); } + finally { clearTimeout(deadline); clearInterval(size); } + }; + let input = '', started = false; const decoder = new TextDecoder('utf-8', { fatal: true }); + try { + for await (const chunk of Bun.stdin.stream()) { + if (started) { if (chunk.byteLength) stop('failed'); continue; } + input += decoder.decode(chunk, { stream: true }); + if (Buffer.byteLength(input) > 16384) { stop('failed'); respond('failed'); } + const end = input.indexOf('\n'); + if (end >= 0) { started = true; if (input.slice(end + 1)) process.exit(2); void run(input.slice(0, end)); input = ''; } + } + stop('cancelled'); if (!started) respond('cancelled'); + } catch { stop('failed'); if (!started) respond('failed'); } +} diff --git a/src/s16/voice/local.ts b/src/s16/voice/local.ts new file mode 100644 index 00000000..0445c96a --- /dev/null +++ b/src/s16/voice/local.ts @@ -0,0 +1,138 @@ +import { createHash } from 'node:crypto'; +import { chmod, lstat, mkdir, mkdtemp, open, readFile, realpath, rm, writeFile } from 'node:fs/promises'; +import { constants } from 'node:fs'; +import { isAbsolute, join, resolve } from 'node:path'; +import { AppError } from '../../errors'; +import { captureWavInfo } from './wav'; + +/** Operator-supplied trusted artifact. A hash pin is not a sandbox or a dependency-tree attestation. */ +export interface VoiceArtifact { path: string; sha256: string } +/** Explicit local-only route: no URL, provider credential, shell command, custom flags or fallback. */ +export interface LocalVoiceConfig { + transcription: { executable: VoiceArtifact; model: VoiceArtifact; language: string }; + synthesis: { kind: 'espeak' | 'say'; executable: VoiceArtifact; voice: string }; + timeoutMs: number; +} +/** Host-only final checks and durable process-start accounting; never supplied by a client/model. */ +export interface VoiceProcessContext { signal: AbortSignal; guard(): void; started(): void } +/** Finite local adapter. Raw audio/text stays in memory/private scratch; errors contain safe codes only. */ +export interface LocalVoiceEngine { + available(): Promise; + transcribe(wav: Uint8Array, context: VoiceProcessContext): Promise; + synthesize(text: string, context: VoiceProcessContext): Promise<{ wav: Uint8Array; durationMs: number }>; + close(): Promise; +} +/** Local synthesis/output bounds, independent of owner-configured quota. */ +export const localSpeechLimits = Object.freeze({ textCharacters: 2000, textBytes: 8192, transcriptBytes: 16384, audioBytes: 4 * 1024 * 1024, durationMs: 60000 }); +const reject = (code: string): never => { throw new AppError(code, 503); }; +const record = (value: unknown, keys: string[]): value is Record => !!value && typeof value === 'object' && !Array.isArray(value) && Object.getPrototypeOf(value) === Object.prototype && Object.keys(value).sort().join() === keys.sort().join(); +const artifact = (value: unknown): value is VoiceArtifact => record(value, ['path', 'sha256']) && typeof value.path === 'string' && isAbsolute(value.path) && resolve(value.path) === value.path && !/[\x00-\x1f\x7f]/.test(value.path) && value.path.length < 2048 && typeof value.sha256 === 'string' && /^[a-f0-9]{64}$/.test(value.sha256); + +/** Dependency-free configuration shape validation; actual regular-file hashes are checked before every process. */ +export function validateLocalVoiceConfig(value: unknown): asserts value is LocalVoiceConfig { + if (!record(value, ['transcription', 'synthesis', 'timeoutMs']) || !record(value.transcription, ['executable', 'model', 'language']) || !record(value.synthesis, ['kind', 'executable', 'voice']) + || !artifact(value.transcription.executable) || !artifact(value.transcription.model) || !artifact(value.synthesis.executable) + || typeof value.transcription.language !== 'string' || !/^[a-z]{2,3}(?:-[A-Za-z]{2,8})?$|^auto$/.test(value.transcription.language) + || !['espeak', 'say'].includes(value.synthesis.kind as string) || typeof value.synthesis.voice !== 'string' || !/^[A-Za-z][A-Za-z0-9 _-]{0,79}$/.test(value.synthesis.voice) + || !Number.isSafeInteger(value.timeoutMs) || (value.timeoutMs as number) < 1 || (value.timeoutMs as number) > 120000) throw new AppError('invalid_voice_config'); +} + +async function verifyArtifact(value: VoiceArtifact, executable: boolean, signal: AbortSignal): Promise { + if (await realpath(value.path) !== value.path) reject('voice_artifact_changed'); + const handle = await open(value.path, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.size < 1 || stat.size > (executable ? 256 * 1024 * 1024 : 1024 * 1024 * 1024) || stat.mode & 0o022 || executable && !(stat.mode & 0o111)) reject('voice_artifact_changed'); + const hash = createHash('sha256'), buffer = Buffer.alloc(65536); let total = 0; + for (;;) { signal.throwIfAborted(); const read = await handle.read(buffer, 0, buffer.length, null); if (!read.bytesRead) break; total += read.bytesRead; if (total > stat.size) reject('voice_artifact_changed'); hash.update(buffer.subarray(0, read.bytesRead)); } + if (total !== stat.size || hash.digest('hex') !== value.sha256) reject('voice_artifact_changed'); + } finally { await handle.close(); } +} + +/** Validate bounded PCM WAV output, allowing the ordinary padded RIFF chunks emitted by local synthesizers. */ +export function speechWavInfo(bytes: Uint8Array): { durationMs: number } { + const invalid = () => reject('voice_audio_invalid'); + if (bytes.length < 46 || bytes.length > localSpeechLimits.audioBytes) return invalid(); + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength), tag = (at: number) => String.fromCharCode(...bytes.subarray(at, at + 4)); + if (tag(0) !== 'RIFF' || tag(8) !== 'WAVE' || view.getUint32(4, true) !== bytes.length - 8) return invalid(); + let at = 12, format: { rate: number; alignment: number } | undefined, size: number | undefined, chunks = 0; + while (at < bytes.length) { + if (++chunks > 32 || at + 8 > bytes.length) return invalid(); + const kind = tag(at), length = view.getUint32(at + 4, true), start = at + 8, end = start + length; + if (end > bytes.length) return invalid(); + if (kind === 'fmt ') { + if (format || length !== 16 && length !== 18 || view.getUint16(start, true) !== 1 || view.getUint16(start + 2, true) !== 1 || view.getUint16(start + 14, true) !== 16) return invalid(); + const rate = view.getUint32(start + 4, true), alignment = view.getUint16(start + 12, true); + if (rate < 8000 || rate > 48000 || alignment !== 2 || view.getUint32(start + 8, true) !== rate * alignment || length === 18 && view.getUint16(start + 16, true) !== 0) return invalid(); + format = { rate, alignment }; + } else if (kind === 'data') { if (size !== undefined || !length || length % 2) return invalid(); size = length; } + else if (!['JUNK', 'LIST', 'fact'].includes(kind)) return invalid(); + at = end + length % 2; + } + if (at !== bytes.length || !format || !size) return invalid(); + const durationMs = size / format.alignment / format.rate * 1000; + if (durationMs > localSpeechLimits.durationMs) reject('voice_audio_limit'); + return { durationMs }; +} + +/** Open an opt-in protected-host adapter; no process/model download or speech starts while checking availability. */ +export async function openLocalVoice(configured: LocalVoiceConfig, scratchPath: string): Promise { + validateLocalVoiceConfig(configured); const config = structuredClone(configured); + if (!isAbsolute(scratchPath) || resolve(scratchPath) !== scratchPath) throw new AppError('invalid_voice_config'); + await mkdir(scratchPath, { recursive: true, mode: 0o700 }); + if (await realpath(scratchPath) !== scratchPath || (await lstat(scratchPath)).mode & 0o077) throw new AppError('invalid_voice_config'); + const shutdown = new AbortController(), jobs = new Set>(); let closed = false; + const verify = async (signal: AbortSignal) => { if (config.synthesis.kind === 'say' && process.platform !== 'darwin') reject('voice_engine_unavailable'); await verifyArtifact(config.transcription.executable, true, signal); await verifyArtifact(config.transcription.model, false, signal); await verifyArtifact(config.synthesis.executable, true, signal); }; + const job = (context: VoiceProcessContext, body: (directory: string, context: VoiceProcessContext) => Promise): Promise => { + const signal = AbortSignal.any([context.signal, shutdown.signal]), check = () => { if (closed || signal.aborted) reject('voice_cancelled'); context.guard(); }; + const running = (async () => { + let directory: string | undefined; + try { check(); await verify(signal); check(); directory = await mkdtemp(join(scratchPath, 'localvoice-')); await chmod(directory, 0o700); check(); return await body(directory, { ...context, signal, guard: check }); } + catch (error) { if (signal.aborted) return reject('voice_cancelled'); if (error instanceof AppError) throw error; return reject('voice_engine_unavailable'); } + finally { if (directory) await rm(directory, { recursive: true, force: true }); } + })(); + jobs.add(running); void running.then(() => jobs.delete(running), () => jobs.delete(running)); return running; + }; + const execute = async (argv: string[], directory: string, output: string, bytes: number, context: VoiceProcessContext) => { + context.guard(); + const worker = Bun.spawn([process.execPath, new URL('./local-worker.ts', import.meta.url).pathname, '--local-voice-worker'], { cwd: directory, stdin: 'pipe', stdout: 'pipe', stderr: 'ignore', env: { PATH: '/usr/bin:/bin', LANG: 'C.UTF-8', LC_ALL: 'C.UTF-8', TMPDIR: directory } }); + const stop = () => { try { worker.stdin.end(); } catch { /* The worker may have exited already. */ } }; + context.signal.addEventListener('abort', stop, { once: true }); + try { + context.guard(); context.started(); context.guard(); + worker.stdin.write(JSON.stringify({ argv, cwd: directory, timeoutMs: config.timeoutMs, output, bytes }) + '\n'); + const [code, raw] = await Promise.all([worker.exited, new Response(worker.stdout).text()]); + context.guard(); + const state = code === 0 && raw.length < 256 ? (JSON.parse(raw) as { state?: string }).state : 'failed'; + if (state !== 'completed') reject(state === 'timeout' ? 'voice_process_timeout' : state === 'limit' ? 'voice_audio_limit' : state === 'cancelled' ? 'voice_cancelled' : 'voice_engine_unavailable'); + } finally { context.signal.removeEventListener('abort', stop); stop(); await worker.exited; } + }; + const boundedFile = async (path: string, max: number) => { const stat = await lstat(path); if (!stat.isFile() || stat.size < 1 || stat.size > max) reject('voice_audio_limit'); const bytes = await readFile(path); if (bytes.length !== stat.size || bytes.length > max) reject('voice_audio_limit'); return bytes; }; + return { + async available() { if (closed) return false; try { await verify(shutdown.signal); return !closed; } catch { return false; } }, + transcribe(input, context) { + const wav = input.slice(); captureWavInfo(wav); + return job(context, async (directory, current) => { + const source = join(directory, 'input.wav'), prefix = join(directory, 'transcript'), output = prefix + '.json'; + await writeFile(source, wav, { mode: 0o600 }); current.guard(); + await execute([config.transcription.executable.path, '--model', config.transcription.model.path, '--file', source, '--language', config.transcription.language, '--no-gpu', '--threads', '2', '--processors', '1', '--output-json', '--output-file', prefix, '--no-prints', '--no-timestamps'], directory, output, 131072, current); + const parsed = JSON.parse((await boundedFile(output, 131072)).toString('utf8')) as { transcription?: { text?: unknown }[] }; + const rows = parsed.transcription; + if (!Array.isArray(rows) || rows.length > 256 || rows.some(row => !row || typeof row.text !== 'string')) return reject('voice_transcript_invalid'); + const text = rows.map(row => row.text).join('').trim(); + if (!text || !text.isWellFormed() || Buffer.byteLength(text) > localSpeechLimits.transcriptBytes || /[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/.test(text)) reject('voice_transcript_invalid'); + current.guard(); return text; + }); + }, + synthesize(text, context) { + if (typeof text !== 'string' || !text.trim() || !text.isWellFormed() || text.length > localSpeechLimits.textCharacters || Buffer.byteLength(text) > localSpeechLimits.textBytes || /[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/.test(text)) throw new AppError('voice_text_limit'); + return job(context, async (directory, current) => { + const input = join(directory, 'reply.txt'), output = join(directory, 'speech.wav'); await writeFile(input, text, { mode: 0o600 }); current.guard(); + const argv = config.synthesis.kind === 'say' ? ['-v', config.synthesis.voice, '-r', '165', '--file-format=WAVE', '--data-format=LEI16@16000', '--channels=1', '-f', input, '-o', output] : ['-v', config.synthesis.voice, '-s', '165', '-f', input, '-w', output]; + await execute([config.synthesis.executable.path, ...argv], directory, output, localSpeechLimits.audioBytes, current); + const wav = await boundedFile(output, localSpeechLimits.audioBytes), info = speechWavInfo(wav); current.guard(); return { wav: Uint8Array.from(wav), ...info }; + }); + }, + async close() { if (!closed) { closed = true; shutdown.abort(); } await Promise.allSettled([...jobs]); }, + }; +} diff --git a/tests/fixtures/s16-voice-parent.ts b/tests/fixtures/s16-voice-parent.ts new file mode 100644 index 00000000..8f2b6775 --- /dev/null +++ b/tests/fixtures/s16-voice-parent.ts @@ -0,0 +1,9 @@ +import { openLocalVoice, type LocalVoiceConfig } from '../../src/s16/voice/local'; +import { encodeCaptureWav } from '../../src/s16/voice/wav'; + +// Disposable controller for the real parent-SIGKILL/CLI-liveness test. No model or microphone is used. +if (process.env.JIT_LOCAL_VOICE_FIXTURE !== '1') throw Error('explicit_voice_fixture_required'); +const input = JSON.parse(await Bun.stdin.text()) as { config: LocalVoiceConfig; scratch: string }; +const engine = await openLocalVoice(input.config, input.scratch); +try { await engine.transcribe(encodeCaptureWav(new Float32Array(1600)), { signal: new AbortController().signal, guard() {}, started() {} }); } +finally { await engine.close(); } diff --git a/tests/s16-voice-local.test.ts b/tests/s16-voice-local.test.ts new file mode 100644 index 00000000..6a1e040c --- /dev/null +++ b/tests/s16-voice-local.test.ts @@ -0,0 +1,106 @@ +import { afterEach, expect, test } from 'bun:test'; +import { createHash } from 'node:crypto'; +import { existsSync, mkdtempSync, readFileSync, readdirSync, realpathSync, rmSync, watch, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { openLocalVoice, speechWavInfo, validateLocalVoiceConfig, type LocalVoiceConfig, type LocalVoiceEngine } from '../src/s16/voice/local'; +import { encodeCaptureWav } from '../src/s16/voice/wav'; +import { AppError } from '../src/errors'; + +const cleanups: (() => void | Promise)[] = []; +afterEach(async () => { for (const close of cleanups.splice(0).reverse()) await close(); }); +const hash = (bytes: string | Uint8Array) => createHash('sha256').update(bytes).digest('hex'); +const wav = () => encodeCaptureWav(new Float32Array(1600)); +const context = (signal = new AbortController().signal) => ({ signal, guard() {}, started() {} }); + +function waitFile(path: string): Promise { + if (existsSync(path)) return Promise.resolve(); + return new Promise((resolve, reject) => { + const watcher = watch(dirname(path), () => { if (existsSync(path)) { clearTimeout(timer); watcher.close(); resolve(); } }); + const timer = setTimeout(() => { watcher.close(); reject(Error('fixture_barrier_timeout')); }, 5000); + if (existsSync(path)) { clearTimeout(timer); watcher.close(); resolve(); } + }); +} + +async function fixture(mode = 'normal', timeoutMs = 5000) { + const root = realpathSync(mkdtempSync(join(tmpdir(), 'jit-voice-local-'))), scratch = join(root, 'scratch'), executable = join(root, 'speech-fixture'), model = join(root, 'model with spaces.bin'), marker = join(root, 'started.json'), release = join(root, 'release'); + cleanups.push(() => rmSync(root, { recursive: true, force: true })); + const source = `#!${process.execPath} +import {existsSync,readFileSync,watch,writeFileSync} from 'node:fs'; +const args=process.argv.slice(2), option=name=>args[args.indexOf(name)+1]; +const mode=${JSON.stringify(mode)}, marker=${JSON.stringify(marker)}, release=${JSON.stringify(release)}; +writeFileSync(marker,JSON.stringify({pid:process.pid,cwd:process.cwd(),args,privateEnv:process.env.JIT_VOICE_PRIVATE_SENTINEL??null,model:args.includes('--model')?readFileSync(option('--model'),'utf8'):null}),{mode:0o600}); +if(mode==='wait') await new Promise(()=>{}); +if(mode==='gate'&&!existsSync(release))await new Promise(resolve=>{const watcher=watch(${JSON.stringify(root)},()=>{if(existsSync(release)){watcher.close();resolve();}});if(existsSync(release)){watcher.close();resolve();}}); +if(mode==='fail'){process.stderr.write('private diagnostic must never escape');process.exit(1);} +if(args.includes('--output-json')) writeFileSync(option('--output-file')+'.json',JSON.stringify({transcription:[{text:' Approve this '},{text:'draft only. '}]}),{mode:0o600}); +else writeFileSync(option('-w'),Buffer.from('52494646240c000057415645666d74201000000001000100803e0000007d00000200100064617461000c0000'+'00'.repeat(3072),'hex'),{mode:0o600}); +`; + writeFileSync(executable, source, { mode: 0o700 }); writeFileSync(model, 'public test model fixture', { mode: 0o600 }); + const config: LocalVoiceConfig = { transcription: { executable: { path: executable, sha256: hash(source) }, model: { path: model, sha256: hash('public test model fixture') }, language: 'auto' }, synthesis: { kind: 'espeak', executable: { path: executable, sha256: hash(source) }, voice: 'en' }, timeoutMs }; + const engine = await openLocalVoice(config, scratch); cleanups.push(() => engine.close()); + return { root, scratch, config, engine, marker, model, release }; +} + +test('actual fixed local CLI receives bounded private files, a clean environment and no command from its transcript', async () => { + const f = await fixture(); process.env.JIT_VOICE_PRIVATE_SENTINEL = 'must_not_be_inherited'; + try { + expect(await f.engine.available()).toBe(true); expect(existsSync(f.marker)).toBe(false); + let started = 0; + expect(await f.engine.transcribe(wav(), { ...context(), started() { started++; } })).toBe('Approve this draft only.'); + const actual = JSON.parse(readFileSync(f.marker, 'utf8')); + expect(actual.privateEnv).toBeNull(); expect(actual.model).toBe('public test model fixture'); expect(actual.args).toContain('--no-gpu'); expect(actual.args).toContain('--threads'); expect(started).toBe(1); + const speech = await f.engine.synthesize('A currently guarded reply.', context()); expect(speech.durationMs).toBe(96); expect(speechWavInfo(speech.wav).durationMs).toBe(96); + expect(readdirSync(f.scratch)).toEqual([]); + } finally { delete process.env.JIT_VOICE_PRIVATE_SENTINEL; } +}); + +test('malformed capture, changed pins and guard revocation refuse before local speech starts', async () => { + const f = await fixture(); let started = 0; + expect(() => f.engine.transcribe(new Uint8Array(), context())).toThrow('voice_audio_invalid'); + await expect(f.engine.transcribe(wav(), { ...context(), guard() { throw Error('revoked'); }, started() { started++; } })).rejects.toThrow('voice_engine_unavailable'); + expect(started).toBe(0); expect(existsSync(f.marker)).toBe(false); + writeFileSync(f.model, 'changed model', { mode: 0o600 }); expect(await f.engine.available()).toBe(false); + await expect(f.engine.transcribe(wav(), context())).rejects.toThrow('voice_artifact_changed'); expect(existsSync(f.marker)).toBe(false); + expect(readdirSync(f.scratch)).toEqual([]); +}); + +test('cancel and close join the actual hung CLI before deleting its input or returning capacity', async () => { + for (const action of ['cancel', 'close']) { + const f = await fixture('wait'), abort = new AbortController(), operation = f.engine.transcribe(wav(), context(abort.signal)); + const result = operation.catch(error => error); await waitFile(f.marker); const pid = JSON.parse(readFileSync(f.marker, 'utf8')).pid; + expect(() => process.kill(pid, 0)).not.toThrow(); + if (action === 'cancel') abort.abort(); else await f.engine.close(); + expect((await result).message).toBe('voice_cancelled'); expect(() => process.kill(pid, 0)).toThrow(); expect(readdirSync(f.scratch)).toEqual([]); + } +}); + +test('the independent supervisor enforces its deadline and only returns safe failure codes', async () => { + const f = await fixture('wait', 1000), pending = f.engine.transcribe(wav(), context()).catch(error => error); + await waitFile(f.marker); const pid = JSON.parse(readFileSync(f.marker, 'utf8')).pid; + expect((await pending).message).toBe('voice_process_timeout'); expect(() => process.kill(pid, 0)).toThrow(); expect(readdirSync(f.scratch)).toEqual([]); + const failed = await fixture('fail'); await expect(failed.engine.transcribe(wav(), context())).rejects.toThrow('voice_engine_unavailable'); +}); + +test('a current owner guard is reacquired after actual speech processing and its safe authentication refusal survives', async () => { + const f = await fixture('gate'); let revoked = false; + const operation = f.engine.transcribe(wav(), { ...context(), guard() { if (revoked) throw new AppError('authentication_required', 401); } }).catch(error => error); + await waitFile(f.marker); revoked = true; writeFileSync(f.release, 'release', { mode: 0o600 }); + expect(await operation).toMatchObject({ code: 'authentication_required', status: 401 }); expect(readdirSync(f.scratch)).toEqual([]); +}); + +test('SIGKILL of the owning controller closes its liveness pipe and the surviving supervisor joins the CLI', async () => { + const f = await fixture('wait'); await f.engine.close(); + const parent = Bun.spawn([process.execPath, join(import.meta.dir, 'fixtures/s16-voice-parent.ts')], { stdin: 'pipe', stdout: 'ignore', stderr: 'ignore', env: { ...process.env, JIT_LOCAL_VOICE_FIXTURE: '1' } }); + parent.stdin.write(JSON.stringify({ config: f.config, scratch: f.scratch })); parent.stdin.end(); + try { + await waitFile(f.marker); const marker = JSON.parse(readFileSync(f.marker, 'utf8')), settled = join(marker.cwd, 'settled.json'); + parent.kill('SIGKILL'); await parent.exited; await waitFile(settled); + expect(JSON.parse(readFileSync(settled, 'utf8'))).toMatchObject({ state: 'cancelled', child: marker.pid }); expect(() => process.kill(marker.pid, 0)).toThrow(); + } finally { parent.kill(); await parent.exited; } +}); + +test('voice configuration has no remote endpoint, arbitrary flags or silently selected route', () => { + expect(() => validateLocalVoiceConfig({ url: 'https://example.com' })).toThrow('invalid_voice_config'); + expect(() => speechWavInfo(new Uint8Array())).toThrow('voice_audio_invalid'); +}); From 3c6d236ea720c45b9331aaa6a97d36cb43b17965 Mon Sep 17 00:00:00 2001 From: Chayapat Sriwattanachote Date: Wed, 18 Nov 2026 11:15:15 +0700 Subject: [PATCH 04/11] feat(voice): retain finite local quota and fence draft recovery across restart --- docs/contracts/s16-voice-v1.md | 4 + src/s16/voice/contracts.ts | 23 ++++ src/s16/voice/local-worker.ts | 2 +- src/s16/voice/local.ts | 47 ++++++- src/s16/voice/service.ts | 190 +++++++++++++++++++++++++++++ tests/fixtures/s16-voice-parent.ts | 22 +++- tests/s16-voice-local.test.ts | 21 ++++ tests/s16-voice-service.test.ts | 111 +++++++++++++++++ 8 files changed, 410 insertions(+), 10 deletions(-) create mode 100644 src/s16/voice/contracts.ts create mode 100644 src/s16/voice/service.ts create mode 100644 tests/s16-voice-service.test.ts diff --git a/docs/contracts/s16-voice-v1.md b/docs/contracts/s16-voice-v1.md index d8e1c4ee..537d2183 100644 --- a/docs/contracts/s16-voice-v1.md +++ b/docs/contracts/s16-voice-v1.md @@ -39,6 +39,10 @@ Raw audio stays in client memory and protected service scratch. The local engine The service owns a separate versioned voice/quota journal with bounded metadata and no audio. It retains original command identities and conservative charges across process loss. A restarted pending local operation becomes terminal/unknown, with no reconstructed execution. Missing/expired draft/audio results are reported honestly; automatic re-recording, re-synthesis or re-submission is forbidden. #423 will extend continuity for confirmed continuous-call turns using the existing durable CHAT receipt, rather than treating a voice response as work acceptance. +The implemented core uses `VoiceService.open` with an exclusive protected journal lease before its engine factory opens scratch. `begin`, `inspect`, `transcribe`, `playback`, `cancel`, `status` and memoized `close` accept trusted current-owner ingress closures; they are not model tools or serialized actor APIs. Client composition remains pending. Command IDs embed issuance time (`voiceRequestId`); first admission permits five minutes of age and thirty seconds of future skew. Retained retries reuse the original binding. Terminal metadata may expire after seven days, but an old timestamped identity still cannot become a new capture. A persisted clock high-water prevents clock rewind from resetting a quota day. At most 4,096 retained operations and 8 MiB / sixteen short-lived response-cache entries are permitted; storage exhaustion refuses new voice work honestly. + +Accepted input duration is known from strict PCM bytes and rounded up to quota milliseconds when transcription starts. Synthesis starts with its conservative maximum and narrows only after a valid actual WAV result. Failed/unknown started processing does not refund the charge. The independent local supervisor retains a private stdin liveness lease; controller loss, cancellation or its own deadline kills and joins its CLI before writing a metadata-only settlement receipt. A reopened engine cleans abandoned scratch only when no command could have been supplied or the matching supervisor receipt proves settlement. An unproven orphan keeps the route unavailable. Runtime composition must place scratch in protected temporary storage outside application state/backups, while the metadata-only journal remains durable. No raw audio/draft is an ordinary export. + Private reply publication is always reacquired through the owning conversation service, including before returning generated audio. Source withdrawal, owner/session revocation, changed result/revision, changed route artifact or generation prevents stale publication. Interruption invalidates client playback without claiming remote work or effects stopped. Audio content, transcripts, quoted commands and model output cannot approve, change scope or configure a route. ## Delivery evidence diff --git a/src/s16/voice/contracts.ts b/src/s16/voice/contracts.ts new file mode 100644 index 00000000..e89be35b --- /dev/null +++ b/src/s16/voice/contracts.ts @@ -0,0 +1,23 @@ +/** Finite owner-client voice support; audio seconds do not measure compute time or provider billing. */ +export const voiceLimits = Object.freeze({ captureMs: 30000, synthesisMs: 60000, transcriptBytes: 16384, responseCacheBytes: 8 * 1024 * 1024, responseCacheMs: 300000, retentionMs: 7 * 86400000, requestAgeMs: 300000, operations: 4096 }); +/** Original request identity embeds its issuance time, so a pruned old command cannot become a fresh capture. */ +export function voiceRequestId(serverTime: number): string { + if (!Number.isSafeInteger(serverTime) || serverTime < 0) throw Error('voice_time_invalid'); + return `v_${serverTime}_${crypto.randomUUID().replaceAll('-', '')}`; +} +/** Explicit local audio budget. Values can narrow, never exceed, published per-operation bounds. */ +export interface VoiceQuotaConfig { dailyMs: number; captureMs: number } +/** Metadata-only local quota; unknown processing retains its conservative charge. All milliseconds round up. */ +export interface VoiceUsage { day: string; unit: 'local_audio_ms'; limitMs: number; chargedMs: number; reservedMs: number; conservativeMs: number } +/** Capture is a private draft operation, never Bot work acceptance or an approval. */ +export interface VoiceCapture { id: string; conversationId: string; state: 'capturing' | 'processing' | 'ready' | 'cancelled' | 'failed' | 'unknown' | 'expired'; expiresAt: number; maxDurationMs: number } +/** Read-only configured capability and current local quota; paths, credentials and model data are absent. */ +export interface VoiceStatus { contract: 's16-voice/v1'; available: boolean; serverTime: number; route: 'local'; limits: VoiceQuotaConfig; usage: VoiceUsage } +/** The owner's explicit Mic intent. It starts no conversation request or model operation. */ +export interface VoiceCaptureCommand { contract: 's16-voice-capture/v1'; requestId: string } +/** Only the exact currently published result can be synthesized; client text is never the source. */ +export interface VoicePlaybackCommand { contract: 's16-voice-playback/v1'; requestId: string; conversationId: string; targetRequestId: string; targetRevision: number; textSha256: string } +/** Editable untrusted draft and its original capture identity. Ordinary Send remains a separate command. */ +export interface VoiceTranscript { contract: 's16-voice-transcript/v1'; capture: VoiceCapture; text: string; usage: VoiceUsage } +/** Bounded audio for one currently guarded result; interruption never implies work/effect cancellation. */ +export interface VoicePlayback { contract: 's16-voice-playback/v1'; id: string; conversationId: string; targetRequestId: string; wavBase64: string; durationMs: number; usage: VoiceUsage } diff --git a/src/s16/voice/local-worker.ts b/src/s16/voice/local-worker.ts index 58bf62ea..30d79c5f 100644 --- a/src/s16/voice/local-worker.ts +++ b/src/s16/voice/local-worker.ts @@ -5,7 +5,7 @@ import { join } from 'node:path'; if (import.meta.main) { if (process.argv[2] !== '--local-voice-worker') process.exit(2); - let child: ReturnType | undefined, reason: string | null = null, finished = false, directory: string | undefined; + let child: ReturnType | undefined, reason: string | null = null, finished = false, directory: string | undefined = process.cwd(); const stop = (code: string) => { reason ??= code; if (child && !finished) { try { child.kill('SIGKILL'); } catch { /* Exit may have won the race. */ } } }; process.on('SIGTERM', () => stop('cancelled')); process.on('SIGINT', () => stop('cancelled')); const respond = (state: string) => { diff --git a/src/s16/voice/local.ts b/src/s16/voice/local.ts index 0445c96a..c8a844a3 100644 --- a/src/s16/voice/local.ts +++ b/src/s16/voice/local.ts @@ -1,6 +1,6 @@ import { createHash } from 'node:crypto'; -import { chmod, lstat, mkdir, mkdtemp, open, readFile, realpath, rm, writeFile } from 'node:fs/promises'; -import { constants } from 'node:fs'; +import { chmod, lstat, mkdir, mkdtemp, open, readdir, realpath, rm, writeFile } from 'node:fs/promises'; +import { constants, writeFileSync } from 'node:fs'; import { isAbsolute, join, resolve } from 'node:path'; import { AppError } from '../../errors'; import { captureWavInfo } from './wav'; @@ -28,6 +28,16 @@ const reject = (code: string): never => { throw new AppError(code, 503); }; const record = (value: unknown, keys: string[]): value is Record => !!value && typeof value === 'object' && !Array.isArray(value) && Object.getPrototypeOf(value) === Object.prototype && Object.keys(value).sort().join() === keys.sort().join(); const artifact = (value: unknown): value is VoiceArtifact => record(value, ['path', 'sha256']) && typeof value.path === 'string' && isAbsolute(value.path) && resolve(value.path) === value.path && !/[\x00-\x1f\x7f]/.test(value.path) && value.path.length < 2048 && typeof value.sha256 === 'string' && /^[a-f0-9]{64}$/.test(value.sha256); +async function boundedFile(path: string, max: number): Promise { + const handle = await open(path, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); if (!stat.isFile() || stat.size < 1 || stat.size > max) reject('voice_audio_limit'); + const buffer = Buffer.alloc(Math.min(65536, max + 1)), chunks: Buffer[] = []; let total = 0; + for (;;) { const read = await handle.read(buffer, 0, Math.min(buffer.length, max + 1 - total), null); if (!read.bytesRead) break; total += read.bytesRead; if (total > max) reject('voice_audio_limit'); chunks.push(Buffer.from(buffer.subarray(0, read.bytesRead))); } + if (total !== stat.size) reject('voice_audio_limit'); return Buffer.concat(chunks); + } finally { await handle.close(); } +} + /** Dependency-free configuration shape validation; actual regular-file hashes are checked before every process. */ export function validateLocalVoiceConfig(value: unknown): asserts value is LocalVoiceConfig { if (!record(value, ['transcription', 'synthesis', 'timeoutMs']) || !record(value.transcription, ['executable', 'model', 'language']) || !record(value.synthesis, ['kind', 'executable', 'voice']) @@ -82,23 +92,50 @@ export async function openLocalVoice(configured: LocalVoiceConfig, scratchPath: await mkdir(scratchPath, { recursive: true, mode: 0o700 }); if (await realpath(scratchPath) !== scratchPath || (await lstat(scratchPath)).mode & 0o077) throw new AppError('invalid_voice_config'); const shutdown = new AbortController(), jobs = new Set>(); let closed = false; + let recovered = false, recovery: Promise | null = null; + const recover = (): Promise => { + if (recovered) return Promise.resolve(true); + return recovery ??= (async () => { + for (const name of await readdir(scratchPath)) { + if (!/^localvoice-[A-Za-z0-9]+$/.test(name)) reject('voice_recovery_pending'); + const directory = join(scratchPath, name), stat = await lstat(directory); + if (!stat.isDirectory() || stat.isSymbolicLink() || stat.mode & 0o077) reject('voice_recovery_pending'); + let intent: { state?: string; worker?: number } | undefined; + try { const data = await boundedFile(join(directory, 'intent.json'), 256); intent = JSON.parse(data.toString('utf8')); } + catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') return false; } + if (intent && intent.state !== 'preparing') { + try { + const data = await boundedFile(join(directory, 'settled.json'), 512); + const settled = JSON.parse(data.toString('utf8')) as { state?: string; worker?: number }; + if (intent.state !== 'worker' || !Number.isSafeInteger(intent.worker) || settled.worker !== intent.worker || !['completed', 'cancelled', 'failed', 'timeout', 'limit'].includes(settled.state ?? '')) return false; + } catch { return false; } + } + // No intent (or preparing) means no command could have been supplied. Otherwise only the + // supervisor's receipt, written after actually joining its child, permits scratch deletion. + await rm(directory, { recursive: true, force: true }); + } + recovered = true; return true; + })().finally(() => { recovery = null; }); + }; const verify = async (signal: AbortSignal) => { if (config.synthesis.kind === 'say' && process.platform !== 'darwin') reject('voice_engine_unavailable'); await verifyArtifact(config.transcription.executable, true, signal); await verifyArtifact(config.transcription.model, false, signal); await verifyArtifact(config.synthesis.executable, true, signal); }; const job = (context: VoiceProcessContext, body: (directory: string, context: VoiceProcessContext) => Promise): Promise => { const signal = AbortSignal.any([context.signal, shutdown.signal]), check = () => { if (closed || signal.aborted) reject('voice_cancelled'); context.guard(); }; const running = (async () => { let directory: string | undefined; - try { check(); await verify(signal); check(); directory = await mkdtemp(join(scratchPath, 'localvoice-')); await chmod(directory, 0o700); check(); return await body(directory, { ...context, signal, guard: check }); } + try { check(); if (!await recover()) reject('voice_recovery_pending'); await verify(signal); check(); directory = await mkdtemp(join(scratchPath, 'localvoice-')); await chmod(directory, 0o700); check(); return await body(directory, { ...context, signal, guard: check }); } catch (error) { if (signal.aborted) return reject('voice_cancelled'); if (error instanceof AppError) throw error; return reject('voice_engine_unavailable'); } finally { if (directory) await rm(directory, { recursive: true, force: true }); } })(); jobs.add(running); void running.then(() => jobs.delete(running), () => jobs.delete(running)); return running; }; const execute = async (argv: string[], directory: string, output: string, bytes: number, context: VoiceProcessContext) => { + await writeFile(join(directory, 'intent.json'), JSON.stringify({ state: 'preparing' }), { mode: 0o600 }); context.guard(); const worker = Bun.spawn([process.execPath, new URL('./local-worker.ts', import.meta.url).pathname, '--local-voice-worker'], { cwd: directory, stdin: 'pipe', stdout: 'pipe', stderr: 'ignore', env: { PATH: '/usr/bin:/bin', LANG: 'C.UTF-8', LC_ALL: 'C.UTF-8', TMPDIR: directory } }); const stop = () => { try { worker.stdin.end(); } catch { /* The worker may have exited already. */ } }; context.signal.addEventListener('abort', stop, { once: true }); try { + writeFileSync(join(directory, 'intent.json'), JSON.stringify({ state: 'worker', worker: worker.pid }), { mode: 0o600 }); context.guard(); context.started(); context.guard(); worker.stdin.write(JSON.stringify({ argv, cwd: directory, timeoutMs: config.timeoutMs, output, bytes }) + '\n'); const [code, raw] = await Promise.all([worker.exited, new Response(worker.stdout).text()]); @@ -107,9 +144,8 @@ export async function openLocalVoice(configured: LocalVoiceConfig, scratchPath: if (state !== 'completed') reject(state === 'timeout' ? 'voice_process_timeout' : state === 'limit' ? 'voice_audio_limit' : state === 'cancelled' ? 'voice_cancelled' : 'voice_engine_unavailable'); } finally { context.signal.removeEventListener('abort', stop); stop(); await worker.exited; } }; - const boundedFile = async (path: string, max: number) => { const stat = await lstat(path); if (!stat.isFile() || stat.size < 1 || stat.size > max) reject('voice_audio_limit'); const bytes = await readFile(path); if (bytes.length !== stat.size || bytes.length > max) reject('voice_audio_limit'); return bytes; }; return { - async available() { if (closed) return false; try { await verify(shutdown.signal); return !closed; } catch { return false; } }, + async available() { if (closed) return false; try { if (!await recover()) return false; await verify(shutdown.signal); return !closed; } catch { return false; } }, transcribe(input, context) { const wav = input.slice(); captureWavInfo(wav); return job(context, async (directory, current) => { @@ -126,6 +162,7 @@ export async function openLocalVoice(configured: LocalVoiceConfig, scratchPath: }, synthesize(text, context) { if (typeof text !== 'string' || !text.trim() || !text.isWellFormed() || text.length > localSpeechLimits.textCharacters || Buffer.byteLength(text) > localSpeechLimits.textBytes || /[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]/.test(text)) throw new AppError('voice_text_limit'); + if (config.synthesis.kind === 'say' && text.includes('[[')) throw new AppError('voice_text_unsupported'); return job(context, async (directory, current) => { const input = join(directory, 'reply.txt'), output = join(directory, 'speech.wav'); await writeFile(input, text, { mode: 0o600 }); current.guard(); const argv = config.synthesis.kind === 'say' ? ['-v', config.synthesis.voice, '-r', '165', '--file-format=WAVE', '--data-format=LEI16@16000', '--channels=1', '-f', input, '-o', output] : ['-v', config.synthesis.voice, '-s', '165', '-f', input, '-w', output]; diff --git a/src/s16/voice/service.ts b/src/s16/voice/service.ts new file mode 100644 index 00000000..edf1a89f --- /dev/null +++ b/src/s16/voice/service.ts @@ -0,0 +1,190 @@ +import { createHash } from 'node:crypto'; +import { AppError } from '../../errors'; +import { DurableJournal } from '../../s15/durable-journal'; +import { canonical, exact, nat } from '../../s15/policy/validation'; +import { identity } from '../../s15/conversation/validation'; +import type { LocalVoiceEngine, VoiceProcessContext } from './local'; +import { captureWavInfo } from './wav'; +import { voiceLimits, type VoiceCapture, type VoiceCaptureCommand, type VoicePlayback, type VoicePlaybackCommand, type VoiceQuotaConfig, type VoiceStatus, type VoiceTranscript, type VoiceUsage } from './contracts'; + +/** Authenticated ingress reacquired after every wait. This host closure is never serialized or offered as a tool. */ +export type VoiceOwner = () => Actor; +/** Existing owning projections supply conversation access and the current guarded result, never cached client text. */ +export interface VoiceServiceOptions { + path: string; installationId: string; create: boolean; limits: VoiceQuotaConfig; + engine(): Promise; + requireOwner(actor: Actor): void; ownerGeneration(): number; + conversation(actor: Actor, id: string): void; + reply(actor: Actor, conversationId: string, requestId: string): { revision: number; text: string }; + now?: () => number; +} +interface Operation { + id: string; kind: 'capture' | 'playback'; conversationId: string; generation: number; commandHash: string; + state: VoiceCapture['state']; createdAt: number; expiresAt: number; day: string; boundMs: number; + reservedMs: number; chargedMs: number; conservative: boolean; started: boolean; inputHash: string | null; inputMs: number | null; +} +interface Active { abort: AbortController; promise: Promise } +const fail = (code: string, status = 409): never => { throw new AppError(code, status); }; +const sha = (value: string | Uint8Array) => createHash('sha256').update(value).digest('hex'); +const day = (at: number) => new Date(at).toISOString().slice(0, 10); +const stamp = (id: string): number => { const match = /^v_(0|[1-9][0-9]{0,15})_[a-f0-9]{32}$/.exec(id); if (!match || !Number.isSafeInteger(Number(match[1]))) return fail('voice_request_invalid', 400); return Number(match[1]); }; + +/** Validate only explicit finite local quotas. Configuration never creates provider/account/billing authority. */ +export function validateVoiceQuota(value: unknown): asserts value is VoiceQuotaConfig { + exact(value, ['dailyMs', 'captureMs']); nat(value.dailyMs, true); nat(value.captureMs, true); + if (value.dailyMs > 3600000 || value.captureMs > voiceLimits.captureMs || value.captureMs > value.dailyMs) fail('invalid_voice_config', 400); +} + +/** Protected owner-only voice journal. It admits no CHAT work and exposes no actor, approval or provider tool. */ +export class VoiceService { + private readonly active = new Map(); + private readonly results = new Map(); + private closed = false; + private closing: Promise | null = null; + private constructor(readonly options: VoiceServiceOptions, private readonly journal: DurableJournal, private readonly engine: LocalVoiceEngine) {} + + /** Acquire the exclusive journal lease before opening local scratch/engine, then retain interrupted usage without replay. */ + static async open(options: VoiceServiceOptions): Promise> { + validateVoiceQuota(options.limits); + options = { ...options, limits: { ...options.limits } }; + const journal = new DurableJournal(options.path, options.installationId, options.create, 0x4a565631); + let engine: LocalVoiceEngine | undefined; + try { + if (options.create) journal.db.exec('CREATE TABLE voice_operations (id TEXT PRIMARY KEY, created_at INTEGER NOT NULL, record_json TEXT NOT NULL)'); + engine = await options.engine(); const service = new VoiceService(options, journal, engine); + journal.tx(() => { + for (const record of service.all()) if (['capturing', 'processing'].includes(record.state)) { record.state = record.started ? 'unknown' : 'expired'; record.reservedMs = 0; service.save(record); } + }); + return service; + } catch (error) { await engine?.close(); journal.close(); throw error; } + } + private all(): Operation[] { return this.journal.db.query<{ record_json: string }, []>('SELECT record_json FROM voice_operations ORDER BY rowid').all().map(row => JSON.parse(row.record_json) as Operation); } + private get(id: string): Operation { stamp(id); const row = this.journal.db.query<{ record_json: string }, [string]>('SELECT record_json FROM voice_operations WHERE id=?').get(id); if (!row) return fail('voice_not_found', 404); return JSON.parse(row.record_json) as Operation; } + private find(id: string): Operation | null { stamp(id); const row = this.journal.db.query<{ record_json: string }, [string]>('SELECT record_json FROM voice_operations WHERE id=?').get(id); return row ? JSON.parse(row.record_json) as Operation : null; } + private save(record: Operation): void { const text = canonical(record); if (Buffer.byteLength(text) > 4096) fail('voice_storage_limit'); this.journal.db.query('INSERT INTO voice_operations VALUES(?,?,?) ON CONFLICT(id) DO UPDATE SET record_json=excluded.record_json').run(record.id, record.createdAt, text); } + private now(): number { + const wall = (this.options.now ?? Date.now)(); if (!Number.isSafeInteger(wall) || wall < 0 || wall > 8640000000000000) return fail('voice_time_invalid'); + const clock = this.journal.db.query<{ json: string }, []>("SELECT json FROM records WHERE id='voice_clock'").get(); + return Math.max(wall, clock ? (JSON.parse(clock.json) as { at: number }).at : 0); + } + private clock(): number { const at = this.now(); this.journal.put('voice_clock', { at }); return at; } + private owner(owner: VoiceOwner, record?: Operation): Actor { + if (this.closed) return fail('voice_unavailable', 503); + const actor = owner(); this.options.requireOwner(actor); + if (record) { if (record.generation !== this.options.ownerGeneration()) return fail('voice_stale', 403); this.options.conversation(actor, record.conversationId); } + return actor; + } + private prune(at: number): void { + // Expired recording reservations sent no CLI command. Processing capacity is released only by its real completion. + for (const record of this.all()) if (record.state === 'capturing' && record.expiresAt <= at) { record.state = 'expired'; record.reservedMs = 0; this.save(record); } + this.journal.db.query("DELETE FROM voice_operations WHERE created_at row.day === today); + return { day: today, unit: 'local_audio_ms', limitMs: this.options.limits.dailyMs, chargedMs: rows.reduce((n, row) => n + row.chargedMs, 0), reservedMs: rows.reduce((n, row) => n + (row.state === 'capturing' && row.expiresAt <= at ? 0 : row.reservedMs), 0), conservativeMs: rows.reduce((n, row) => n + (row.conservative ? row.chargedMs : 0), 0) }; + } + private capture(record: Operation): VoiceCapture { return { id: record.id, conversationId: record.conversationId, state: record.state, expiresAt: record.expiresAt, maxDurationMs: record.boundMs }; } + private cached(id: string, owner: VoiceOwner): T { + const record = this.get(id); this.owner(owner, record); + const result = this.results.get(id); if (!result || result.until <= this.now()) return fail(record.state === 'unknown' ? 'voice_outcome_unknown' : 'voice_result_expired'); + return { ...structuredClone(result.value), usage: this.usageAt(this.now()) } as T; + } + private cache(id: string, value: VoiceTranscript | VoicePlayback): void { + const bytes = Buffer.byteLength(JSON.stringify(value)); if (bytes > voiceLimits.responseCacheBytes) fail('voice_audio_limit'); + this.results.set(id, { value: structuredClone(value), bytes, until: this.now() + voiceLimits.responseCacheMs }); + let total = [...this.results.values()].reduce((n, entry) => n + entry.bytes, 0); + for (const [key, entry] of this.results) { if (total <= voiceLimits.responseCacheBytes && this.results.size <= 16) break; this.results.delete(key); total -= entry.bytes; } + } + private fresh(id: string, at: number): void { const issued = stamp(id); if (issued > at + 30000 || issued < at - voiceLimits.requestAgeMs) fail('voice_request_expired'); } + private capacity(at: number, upper: number): void { + this.prune(at); + if (this.all().some(row => ['capturing', 'processing'].includes(row.state))) fail('voice_busy'); + if (this.all().length >= voiceLimits.operations) fail('voice_storage_limit'); + const usage = this.usageAt(at); if (upper > usage.limitMs - usage.chargedMs - usage.reservedMs) fail('voice_budget_exhausted'); + } + + /** Metadata-only status; health inspection never starts capture, speech or Bot work. */ + async status(owner: VoiceOwner): Promise { + this.owner(owner); const available = await this.engine.available(); this.owner(owner); + const at = this.now(); return { contract: 's16-voice/v1', available, serverTime: at, route: 'local', limits: { ...this.options.limits }, usage: this.usageAt(at) }; + } + /** Reserve a bounded draft. Retried/expired identities cannot silently create another capture. */ + async begin(owner: VoiceOwner, conversationId: string, input: VoiceCaptureCommand): Promise { + exact(input, ['contract', 'requestId']); if (input.contract !== 's16-voice-capture/v1') fail('voice_request_invalid', 400); identity(conversationId); stamp(input.requestId); + const command = structuredClone(input), hash = sha(canonical({ ...command, conversationId })); + this.options.conversation(this.owner(owner), conversationId); + if (!await this.engine.available()) fail('voice_unavailable', 503); + return this.journal.tx(() => { + this.options.conversation(this.owner(owner), conversationId); const at = this.clock(), old = this.find(command.requestId); + if (old) { if (old.commandHash !== hash) fail('request_id_conflict'); this.owner(owner, old); this.prune(at); return this.capture(this.get(old.id)); } + this.fresh(command.requestId, at); this.capacity(at, this.options.limits.captureMs); + const record: Operation = { id: command.requestId, kind: 'capture', conversationId, generation: this.options.ownerGeneration(), commandHash: hash, state: 'capturing', createdAt: at, expiresAt: at + this.options.limits.captureMs + 15000, day: day(at), boundMs: this.options.limits.captureMs, reservedMs: this.options.limits.captureMs, chargedMs: 0, conservative: false, started: false, inputHash: null, inputMs: null }; + this.save(record); return this.capture(record); + }); + } + /** Inspect current capture metadata without exposing its transcript/audio or admitting new work. */ + inspect(owner: VoiceOwner, id: string): VoiceCapture { this.owner(owner); const record = this.get(id); this.owner(owner, record); return { ...this.capture(record), state: record.state === 'capturing' && record.expiresAt <= this.now() ? 'expired' : record.state }; } + + private run(owner: VoiceOwner, record: Operation, signal: AbortSignal, body: (context: VoiceProcessContext) => Promise, finalGuard?: () => void): Promise { + const abort = new AbortController(), stop = () => abort.abort(); signal.addEventListener('abort', stop, { once: true }); if (signal.aborted) stop(); + const guard = () => { if (abort.signal.aborted) fail('voice_cancelled'); const current = this.get(record.id); this.owner(owner, current); if (current.state !== 'processing') fail('voice_cancelled'); }; + const promise = Promise.resolve().then(async () => { + try { + guard(); const result = await body({ signal: abort.signal, guard, started: () => this.journal.tx(() => { guard(); const current = this.get(record.id); if (current.started) fail('voice_process_replay'); current.started = true; current.reservedMs = 0; current.chargedMs = current.kind === 'capture' ? current.inputMs! : current.boundMs; current.conservative = current.kind === 'playback'; this.save(current); }) }); + if (abort.signal.aborted) fail('voice_cancelled'); + const current = this.get(record.id); this.owner(owner, current); if (current.state !== 'ready') fail('voice_cancelled'); + finalGuard?.(); this.cache(record.id, result); return structuredClone(result); + } catch (error) { + this.journal.tx(() => { const current = this.get(record.id); current.state = abort.signal.aborted ? 'cancelled' : current.started ? 'unknown' : 'failed'; current.reservedMs = 0; this.save(current); }); + this.results.delete(record.id); throw error; + } finally { signal.removeEventListener('abort', stop); this.active.delete(record.id); } + }); + this.active.set(record.id, { abort, promise }); return promise; + } + + /** Strict raw WAV is snapshotted/bound once; cancellation or a lost result never replays transcription. */ + transcribe(owner: VoiceOwner, id: string, input: Uint8Array, signal: AbortSignal): Promise { + this.owner(owner); + const wav = input.slice(), info = captureWavInfo(wav), hash = sha(wav); let record = this.get(id); this.owner(owner, record); + if (record.kind !== 'capture' || info.durationMs > record.boundMs) return fail('voice_audio_limit', 400); + if (record.inputHash && record.inputHash !== hash) return fail('request_id_conflict'); + const active = this.active.get(id); if (active) return active.promise.then(() => this.cached(id, owner)); + if (record.state !== 'capturing') return Promise.resolve(this.cached(id, owner)); + record = this.journal.tx(() => { const current = this.get(id), at = this.clock(); this.owner(owner, current); if (at >= current.expiresAt) fail('voice_capture_expired'); current.state = 'processing'; current.inputHash = hash; current.inputMs = Math.ceil(info.durationMs); this.save(current); return current; }); + return this.run(owner, record, signal, async context => { + const text = await this.engine.transcribe(wav, context); context.guard(); + return this.journal.tx(() => { context.guard(); const current = this.get(id); current.state = 'ready'; current.reservedMs = 0; current.chargedMs = current.inputMs!; current.conservative = false; this.save(current); return { contract: 's16-voice-transcript/v1', capture: this.capture(current), text, usage: this.usageAt(this.clock()) }; }); + }); + } + /** Reacquire exact current result before processing and before publishing bounded local reply audio. */ + async playback(owner: VoiceOwner, input: VoicePlaybackCommand, signal: AbortSignal): Promise { + exact(input, ['contract', 'requestId', 'conversationId', 'targetRequestId', 'targetRevision', 'textSha256']); + if (input.contract !== 's16-voice-playback/v1' || !/^[a-f0-9]{64}$/.test(input.textSha256)) fail('voice_request_invalid', 400); + identity(input.conversationId); identity(input.targetRequestId); nat(input.targetRevision, true); stamp(input.requestId); + const command = structuredClone(input), hash = sha(canonical(command)); + const reply = () => { const actor = this.owner(owner), value = this.options.reply(actor, command.conversationId, command.targetRequestId); if (value.revision !== command.targetRevision || sha(value.text) !== command.textSha256) fail('voice_result_changed'); return value.text; }; + const text = reply(), existing = this.find(command.requestId); + if (existing) { if (existing.commandHash !== hash) fail('request_id_conflict'); const active = this.active.get(existing.id); if (active) await active.promise; reply(); return this.cached(existing.id, owner); } + if (!await this.engine.available()) fail('voice_unavailable', 503); + const admitted = this.journal.tx(() => { + const old = this.find(command.requestId); + if (old) { if (old.commandHash !== hash) fail('request_id_conflict'); return { record: old, existing: true }; } + reply(); const at = this.clock(); this.fresh(command.requestId, at); this.capacity(at, voiceLimits.synthesisMs); + const value: Operation = { id: command.requestId, kind: 'playback', conversationId: command.conversationId, generation: this.options.ownerGeneration(), commandHash: hash, state: 'processing', createdAt: at, expiresAt: at + 120000, day: day(at), boundMs: voiceLimits.synthesisMs, reservedMs: voiceLimits.synthesisMs, chargedMs: 0, conservative: false, started: false, inputHash: command.textSha256, inputMs: null }; this.save(value); return { record: value, existing: false }; + }); + const record = admitted.record; + if (admitted.existing) { const active = this.active.get(record.id); if (active) await active.promise; reply(); return this.cached(record.id, owner); } + return this.run(owner, record, signal, async context => { + const audio = await this.engine.synthesize(text, { ...context, guard() { context.guard(); reply(); } }); context.guard(); reply(); + return this.journal.tx(() => { context.guard(); reply(); const current = this.get(record.id); current.state = 'ready'; current.reservedMs = 0; current.chargedMs = Math.ceil(audio.durationMs); current.conservative = false; this.save(current); return { contract: 's16-voice-playback/v1', id: current.id, conversationId: command.conversationId, targetRequestId: command.targetRequestId, wavBase64: Buffer.from(audio.wav).toString('base64'), durationMs: Math.ceil(audio.durationMs), usage: this.usageAt(this.clock()) }; }); + }, reply); + } + /** Cancel fences private drafts/playback only; it never stops or approves underlying CHAT work. */ + async cancel(owner: VoiceOwner, id: string): Promise { + this.owner(owner); const active = this.active.get(id); active?.abort.abort(); if (active) await active.promise.catch(() => {}); + return this.journal.tx(() => { this.owner(owner); this.clock(); const record = this.get(id); record.state = 'cancelled'; record.reservedMs = 0; this.results.delete(id); this.save(record); return this.capture(record); }); + } + /** Native engine closure joins all children before journal/capacity custody closes. */ + close(): Promise { return this.closing ??= (async () => { this.closed = true; for (const active of this.active.values()) active.abort.abort(); await this.engine.close(); await Promise.allSettled([...this.active.values()].map(value => value.promise)); this.results.clear(); this.journal.close(); })(); } +} diff --git a/tests/fixtures/s16-voice-parent.ts b/tests/fixtures/s16-voice-parent.ts index 8f2b6775..29bd61eb 100644 --- a/tests/fixtures/s16-voice-parent.ts +++ b/tests/fixtures/s16-voice-parent.ts @@ -1,9 +1,23 @@ import { openLocalVoice, type LocalVoiceConfig } from '../../src/s16/voice/local'; import { encodeCaptureWav } from '../../src/s16/voice/wav'; +import { VoiceService } from '../../src/s16/voice/service'; +import { voiceRequestId } from '../../src/s16/voice/contracts'; +import { writeFileSync } from 'node:fs'; +import { join } from 'node:path'; // Disposable controller for the real parent-SIGKILL/CLI-liveness test. No model or microphone is used. if (process.env.JIT_LOCAL_VOICE_FIXTURE !== '1') throw Error('explicit_voice_fixture_required'); -const input = JSON.parse(await Bun.stdin.text()) as { config: LocalVoiceConfig; scratch: string }; -const engine = await openLocalVoice(input.config, input.scratch); -try { await engine.transcribe(encodeCaptureWav(new Float32Array(1600)), { signal: new AbortController().signal, guard() {}, started() {} }); } -finally { await engine.close(); } +const input = JSON.parse(await Bun.stdin.text()) as { config: LocalVoiceConfig; scratch: string; journal?: string }; +if (input.journal) { + const service = await VoiceService.open({ path: input.journal, installationId: 'fixture', create: true, limits: { dailyMs: 120000, captureMs: 30000 }, now: () => 1000000, + ownerGeneration: () => 1, requireOwner() {}, conversation() {}, reply() { throw Error('not_used'); }, engine: () => openLocalVoice(input.config, input.scratch) }); + try { + const capture = await service.begin(() => ({}), 'conversation', { contract: 's16-voice-capture/v1', requestId: voiceRequestId(1000000) }); + writeFileSync(join(input.scratch, '..', 'capture.json'), JSON.stringify(capture), { mode: 0o600 }); + await service.transcribe(() => ({}), capture.id, encodeCaptureWav(new Float32Array(1600)), new AbortController().signal); + } finally { await service.close(); } +} else { + const engine = await openLocalVoice(input.config, input.scratch); + try { await engine.transcribe(encodeCaptureWav(new Float32Array(1600)), { signal: new AbortController().signal, guard() {}, started() {} }); } + finally { await engine.close(); } +} diff --git a/tests/s16-voice-local.test.ts b/tests/s16-voice-local.test.ts index 6a1e040c..44ad2bac 100644 --- a/tests/s16-voice-local.test.ts +++ b/tests/s16-voice-local.test.ts @@ -6,6 +6,7 @@ import { dirname, join } from 'node:path'; import { openLocalVoice, speechWavInfo, validateLocalVoiceConfig, type LocalVoiceConfig, type LocalVoiceEngine } from '../src/s16/voice/local'; import { encodeCaptureWav } from '../src/s16/voice/wav'; import { AppError } from '../src/errors'; +import { VoiceService } from '../src/s16/voice/service'; const cleanups: (() => void | Promise)[] = []; afterEach(async () => { for (const close of cleanups.splice(0).reverse()) await close(); }); @@ -97,6 +98,8 @@ test('SIGKILL of the owning controller closes its liveness pipe and the survivin await waitFile(f.marker); const marker = JSON.parse(readFileSync(f.marker, 'utf8')), settled = join(marker.cwd, 'settled.json'); parent.kill('SIGKILL'); await parent.exited; await waitFile(settled); expect(JSON.parse(readFileSync(settled, 'utf8'))).toMatchObject({ state: 'cancelled', child: marker.pid }); expect(() => process.kill(marker.pid, 0)).toThrow(); + const reopened = await openLocalVoice(f.config, f.scratch); + try { expect(await reopened.available()).toBe(true); expect(readdirSync(f.scratch)).toEqual([]); } finally { await reopened.close(); } } finally { parent.kill(); await parent.exited; } }); @@ -104,3 +107,21 @@ test('voice configuration has no remote endpoint, arbitrary flags or silently se expect(() => validateLocalVoiceConfig({ url: 'https://example.com' })).toThrow('invalid_voice_config'); expect(() => speechWavInfo(new Uint8Array())).toThrow('voice_audio_invalid'); }); + +test('a real controller crash retains its started quota and cannot replay the original transcription after journal recovery', async () => { + const f = await fixture('wait'); await f.engine.close(); const journal = join(f.root, 'voice.sqlite'); + const parent = Bun.spawn([process.execPath, join(import.meta.dir, 'fixtures/s16-voice-parent.ts')], { stdin: 'pipe', stdout: 'ignore', stderr: 'ignore', env: { ...process.env, JIT_LOCAL_VOICE_FIXTURE: '1' } }); + parent.stdin.write(JSON.stringify({ config: f.config, scratch: f.scratch, journal })); parent.stdin.end(); + try { + await waitFile(f.marker); const marker = JSON.parse(readFileSync(f.marker, 'utf8')); parent.kill('SIGKILL'); await parent.exited; await waitFile(join(marker.cwd, 'settled.json')); + const capture = JSON.parse(readFileSync(join(f.root, 'capture.json'), 'utf8')); + const service = await VoiceService.open({ path: journal, installationId: 'fixture', create: false, limits: { dailyMs: 120000, captureMs: 30000 }, now: () => 1000000, + ownerGeneration: () => 1, requireOwner() {}, conversation() {}, reply() { throw Error('not_used'); }, engine: () => openLocalVoice(f.config, f.scratch) }); + try { + expect(service.inspect(() => ({}), capture.id).state).toBe('unknown'); + expect((await service.status(() => ({}))).usage).toMatchObject({ chargedMs: 100, reservedMs: 0 }); + expect(() => service.transcribe(() => ({}), capture.id, wav(), new AbortController().signal)).toThrow('voice_outcome_unknown'); + expect(JSON.parse(readFileSync(f.marker, 'utf8')).pid).toBe(marker.pid); expect(() => process.kill(marker.pid, 0)).toThrow(); expect(readdirSync(f.scratch)).toEqual([]); + } finally { await service.close(); } + } finally { parent.kill(); await parent.exited; } +}); diff --git a/tests/s16-voice-service.test.ts b/tests/s16-voice-service.test.ts new file mode 100644 index 00000000..2952b933 --- /dev/null +++ b/tests/s16-voice-service.test.ts @@ -0,0 +1,111 @@ +import { afterEach, expect, test } from 'bun:test'; +import { createHash } from 'node:crypto'; +import { mkdtempSync, realpathSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { AppError } from '../src/errors'; +import { VoiceService } from '../src/s16/voice/service'; +import { voiceLimits, voiceRequestId, type VoicePlaybackCommand } from '../src/s16/voice/contracts'; +import type { LocalVoiceEngine, VoiceProcessContext } from '../src/s16/voice/local'; +import { encodeCaptureWav } from '../src/s16/voice/wav'; + +const cleanup: (() => void | Promise)[] = []; +afterEach(async () => { for (const close of cleanup.splice(0).reverse()) await close(); }); +const signal = () => new AbortController().signal; +const wav = () => encodeCaptureWav(new Float32Array(1600)); +const hash = (text: string) => createHash('sha256').update(text).digest('hex'); +async function gate(context: VoiceProcessContext, pending: Promise | null) { + if (!pending) return; + let stop!: () => void; + const abort = new Promise((_, reject) => { stop = () => reject(new AppError('voice_cancelled')); context.signal.addEventListener('abort', stop, { once: true }); if (context.signal.aborted) stop(); }); + try { await Promise.race([pending, abort]); } finally { context.signal.removeEventListener('abort', stop); } +} + +async function fixture(dailyMs = 120000, captureMs = 30000) { + const root = realpathSync(mkdtempSync(join(tmpdir(), 'jit-voice-service-'))), owner = Object.freeze({}); + cleanup.push(() => rmSync(root, { recursive: true, force: true })); + let at = 1000000, generation = 1, allowed = true, source = true, available = true, text = 'A currently published result.', revision = 1; + let transcriptions = 0, syntheses = 0, recording: Promise | null = null, speaking: Promise | null = null, availability: Promise | null = null, closeBarrier: Promise | null = null, beforeStart = false; + const entered = Promise.withResolvers(), speechEntered = Promise.withResolvers(); + const makeEngine = (): LocalVoiceEngine => ({ + async available() { await availability; return available; }, + async transcribe(_wav, context) { if (beforeStart) throw new AppError('voice_unavailable'); context.guard(); context.started(); transcriptions++; entered.resolve(); await gate(context, recording); context.guard(); return 'Approve this draft only.'; }, + async synthesize(_text, context) { context.guard(); context.started(); syntheses++; speechEntered.resolve(); await gate(context, speaking); context.guard(); return { wav: wav(), durationMs: 100 }; }, + async close() { await closeBarrier; }, + }); + const options = (create: boolean) => ({ path: join(root, 'voice.sqlite'), installationId: 'fixture', create, limits: { dailyMs, captureMs }, engine: async () => makeEngine(), + now: () => at, ownerGeneration: () => generation, requireOwner(actor: object) { if (actor !== owner || !allowed) throw new AppError('authentication_required', 401); }, conversation(_actor: object, id: string) { if (id !== 'conversation') throw new AppError('outside_scope'); }, + reply() { if (!source) throw new AppError('source_withdrawn', 403); return { text, revision }; } }); + let service = await VoiceService.open(options(true)); cleanup.push(() => service.close()); + const f = { owner: () => owner, entered: entered.promise, speechEntered: speechEntered.promise, get service() { return service; }, get transcriptions() { return transcriptions; }, get syntheses() { return syntheses; }, + set recording(value: Promise | null) { recording = value; }, set speaking(value: Promise | null) { speaking = value; }, set availability(value: Promise | null) { availability = value; }, set closeBarrier(value: Promise | null) { closeBarrier = value; }, + set beforeStart(value: boolean) { beforeStart = value; }, set allowed(value: boolean) { allowed = value; }, set source(value: boolean) { source = value; }, set generation(value: number) { generation = value; }, set at(value: number) { at = value; }, + command: () => ({ contract: 's16-voice-capture/v1' as const, requestId: voiceRequestId(at) }), + playback: (): VoicePlaybackCommand => ({ contract: 's16-voice-playback/v1', requestId: voiceRequestId(at), conversationId: 'conversation', targetRequestId: 'request', targetRevision: revision, textSha256: hash(text) }), + async reopen() { await service.close(); service = await VoiceService.open(options(false)); }, + }; return f; +} + +test('Mic reserve/retry/cancel creates only a private draft; status starts no speech', async () => { + const f = await fixture(), command = f.command(); + expect((await f.service.status(f.owner)).usage).toMatchObject({ chargedMs: 0, reservedMs: 0 }); + const capture = await f.service.begin(f.owner, 'conversation', command); + expect(await f.service.begin(f.owner, 'conversation', command)).toEqual(capture); + expect((await f.service.status(f.owner)).usage.reservedMs).toBe(30000); expect(f.transcriptions).toBe(0); expect(f.syntheses).toBe(0); + expect((await f.service.cancel(f.owner, capture.id)).state).toBe('cancelled'); + expect((await f.service.status(f.owner)).usage).toMatchObject({ chargedMs: 0, reservedMs: 0 }); +}); + +test('one snapshotted WAV has one transcription across concurrent retry, with no approval or work admission', async () => { + const f = await fixture(), release = Promise.withResolvers(); f.recording = release.promise; + const capture = await f.service.begin(f.owner, 'conversation', f.command()), input = wav(); + const first = f.service.transcribe(f.owner, capture.id, input, signal()); await f.entered; + const second = f.service.transcribe(f.owner, capture.id, input, signal()); input.fill(0); release.resolve(); + const result = await first; expect(await second).toEqual(result); expect(result.text).toBe('Approve this draft only.'); expect(result.capture.state).toBe('ready'); + expect(f.transcriptions).toBe(1); expect(result.usage).toMatchObject({ chargedMs: 100, reservedMs: 0, conservativeMs: 0 }); + const changed = wav(); changed[44] = 1; expect(() => f.service.transcribe(f.owner, capture.id, changed, signal())).toThrow('request_id_conflict'); + f.generation = 2; expect(() => f.service.transcribe(f.owner, capture.id, wav(), signal())).toThrow('voice_stale'); +}); + +test('permission/capture expiry releases an unused reservation and a pruned old identity never becomes a fresh Mic intent', async () => { + const f = await fixture(), command = f.command(), capture = await f.service.begin(f.owner, 'conversation', command); + f.at = capture.expiresAt; + expect(f.service.inspect(f.owner, capture.id).state).toBe('expired'); expect((await f.service.status(f.owner)).usage.reservedMs).toBe(0); + expect(() => f.service.transcribe(f.owner, capture.id, wav(), signal())).toThrow('voice_capture_expired'); + f.at = capture.expiresAt + voiceLimits.retentionMs + 1; + const fresh = await f.service.begin(f.owner, 'conversation', f.command()); await f.service.cancel(f.owner, fresh.id); + await expect(f.service.begin(f.owner, 'conversation', command)).rejects.toThrow('voice_request_expired'); expect(f.transcriptions).toBe(0); +}); + +test('local input budget persists across restart and clock rewind cannot reset the current quota day', async () => { + const f = await fixture(100, 100), capture = await f.service.begin(f.owner, 'conversation', f.command()); + await f.service.transcribe(f.owner, capture.id, wav(), signal()); await f.reopen(); + await expect(f.service.begin(f.owner, 'conversation', f.command())).rejects.toThrow('voice_budget_exhausted'); + f.at = 86400000 + 1000000; const next = await f.service.begin(f.owner, 'conversation', f.command()); await f.service.cancel(f.owner, next.id); + f.at = 1000000; const current = await f.service.status(f.owner); expect(current.usage.day).toBe('1970-01-02'); expect(current.serverTime).toBe(87400000); +}); + +test('withdrawal during local synthesis prevents audio publication and retains conservative usage across restart', async () => { + const f = await fixture(), release = Promise.withResolvers(); f.speaking = release.promise; + const command = f.playback(), operation = f.service.playback(f.owner, command, signal()).catch(error => error); + await f.speechEntered; f.source = false; release.resolve(); + expect(await operation).toMatchObject({ code: 'source_withdrawn' }); + expect((await f.service.status(f.owner)).usage).toMatchObject({ chargedMs: 60000, reservedMs: 0, conservativeMs: 60000 }); + f.source = true; await f.reopen(); await expect(f.service.playback(f.owner, command, signal())).rejects.toThrow('voice_outcome_unknown'); expect(f.syntheses).toBe(1); +}); + +test('two playback calls racing route health reuse one identity and one actual synthesis', async () => { + const f = await fixture(), available = Promise.withResolvers(), release = Promise.withResolvers(); f.availability = available.promise; f.speaking = release.promise; + const command = f.playback(), one = f.service.playback(f.owner, command, signal()), two = f.service.playback(f.owner, command, signal()); + available.resolve(); await f.speechEntered; release.resolve(); + expect(await two).toEqual(await one); expect(f.syntheses).toBe(1); expect((await f.service.status(f.owner)).usage.chargedMs).toBe(100); +}); + +test('failure before process start releases quota and repeated close callers both wait for physical settlement', async () => { + const f = await fixture(), capture = await f.service.begin(f.owner, 'conversation', f.command()); f.beforeStart = true; + await expect(f.service.transcribe(f.owner, capture.id, wav(), signal())).rejects.toThrow('voice_unavailable'); + expect((await f.service.status(f.owner)).usage).toMatchObject({ chargedMs: 0, reservedMs: 0 }); + const closed = Promise.withResolvers(); f.closeBarrier = closed.promise; + const first = f.service.close(), second = f.service.close(); let finished = false; void second.then(() => { finished = true; }); + await Promise.resolve(); expect(finished).toBe(false); closed.resolve(); await Promise.all([first, second]); expect(finished).toBe(true); +}); From a5b783ff1f7443709a2ff4abaa38a30c218b6119 Mon Sep 17 00:00:00 2001 From: Chayapat Sriwattanachote Date: Wed, 18 Nov 2026 11:33:42 +0700 Subject: [PATCH 05/11] feat(voice): bind local drafts and guarded reply playback to shared client ingress --- src/s15/composition.ts | 25 +++++++--- src/s15/main.ts | 1 + src/s15/runtime-config.ts | 9 ++++ src/s16/client/client.ts | 57 +++++++++++++++++++++ src/s16/client/contracts.ts | 2 + src/s16/client/index.ts | 1 + src/s16/http.ts | 34 ++++++++++++- src/s16/voice/service.ts | 14 ++++++ tests/s16-voice-client.test.ts | 90 ++++++++++++++++++++++++++++++++++ 9 files changed, 225 insertions(+), 8 deletions(-) create mode 100644 tests/s16-voice-client.test.ts diff --git a/src/s15/composition.ts b/src/s15/composition.ts index 2bc4ba4a..3f8e586b 100644 --- a/src/s15/composition.ts +++ b/src/s15/composition.ts @@ -1,10 +1,14 @@ import { SkillDirectory } from "../s16/skills/service"; +import { VoiceService } from "../s16/voice/service"; +import { openLocalVoice, type LocalVoiceConfig, type LocalVoiceEngine } from "../s16/voice/local"; +import type { VoiceQuotaConfig } from "../s16/voice/contracts"; +import { tmpdir } from 'node:os'; import { createSkillRuntime, skillVmSessions } from "../s16/skills/runtime"; import { skillDisclosure } from "../s16/skills/disclosure"; import { disclosureMemory } from "./disclosure/data-memory"; import { ComputerContinuations } from "../s16/computers/continuation"; import { websiteHostSensitive } from "./browser/sites"; -import { fsyncSync, lstatSync, mkdirSync, openSync, closeSync, readFileSync, renameSync, rmSync, writeFileSync } from 'node:fs'; +import { fsyncSync, lstatSync, mkdirSync, openSync, closeSync, readFileSync, realpathSync, renameSync, rmSync, writeFileSync } from 'node:fs'; import { isAbsolute, join, resolve } from 'node:path'; import { createS16ModelsPort } from '../s16/models'; import { createS16ClientHandler } from '../s16/http'; @@ -39,7 +43,7 @@ import { createHostCredentialLifecycle, type HostAuthFetch, type HostClock, type import { AppError } from '../errors'; import { openPolicyService, type ActionAdapter, type AuthenticatedOwner, type CatalogPort, type OwnerActor, type PolicyInput, type ReadHandler, type Ref, type S15PolicyService, type SubscriptionModels, type WorkActor } from './policy'; import { canonical, digest, equal, unknownUsage } from './policy/validation'; -import { ConversationApplication, ConversationStore, conversationRequestView, createConversationHandler, createConversationPolicy, createConversationVmTools, type ConversationPolicyWork, type ConversationToolPort } from './conversation'; +import { ConversationApplication, ConversationStore, conversationOwnerView, conversationRequestView, createConversationHandler, createConversationPolicy, createConversationVmTools, type ConversationPolicyWork, type ConversationToolPort } from './conversation'; import { WorkingVmService, VmEnrollment, VmManagers, VmPolicyIntegration, VmPool, VmStore, LocalVmProvisioning, createVmApi, isVmLocalManagement, placeWorkingVm, probeLocalVmHost, startLocalVmControl, type LocalVmControlOptions, type WorkingVmOptions } from './vm'; import { privateLocalVmResources } from './vm/local'; import { registerProtectedBrowser, type BrowserOptions, type BrowserRegistration } from './browser'; @@ -106,6 +110,9 @@ export interface S15CompositionOptions { connections?: { vault: import('../exec/connections').ConnectionVault; network?: SourceNetworkPorts }; /** Explicit SKILL16-01 pinned inert packages and guarded working-computer use; requires DATA15. */ skills?: { network?: SourceNetworkPorts }; + /** Explicit local owner voice, outside Bot execution. Scratch defaults outside state/backups; injected engine/scratch + * are trusted test/host seams, never runtime-client fields. A configured route changes no standing policy. */ + voice?: { local: LocalVoiceConfig; quota: VoiceQuotaConfig; engine?: () => Promise; scratchDirectory?: string }; /** Selected host transition. Status is read-only; its activity guard fences work and records released * activity durably. Composition cannot activate/release or import old authority through this port. */ adoption?: { service: Pick; authenticate(owner: OwnerActor): import('./adoption').AdoptionOwner }; @@ -175,7 +182,7 @@ export async function composeS15(options: S15CompositionOptions) { let skillCopyCurrent: (vmId: string, resource: Ref) => boolean = () => false; if (options.skills && options.disclosure) options = { ...options, disclosure: skillDisclosure(options.disclosure, { directory: () => skills, ownerGeneration: options.owner.generation, computer: (work, vmId) => skillComputer(work, vmId), copyCurrent: (vmId, resource) => skillCopyCurrent(vmId, resource), work: { ...s15DefaultRefs.work }, interpretation: { ...s15DefaultRefs.interpretation }, now: options.clock?.now ?? Date.now }) }; const root = options.directory, manifestPath = join(root, 'composition.json'); - const storeNames = ['conversation.sqlite', 'policy.sqlite', 'model.sqlite', 'work.sqlite', 'vm.sqlite', 'team.sqlite', 'memory.sqlite', ...(options.disclosure ? ['disclosure.sqlite'] : []), ...(options.browser ? ['browser.sqlite'] : []), ...(options.github || webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : [])]; + const storeNames = ['conversation.sqlite', 'policy.sqlite', 'model.sqlite', 'work.sqlite', 'vm.sqlite', 'team.sqlite', 'memory.sqlite', ...(options.disclosure ? ['disclosure.sqlite'] : []), ...(options.browser ? ['browser.sqlite'] : []), ...(options.github || webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : []), ...(options.voice ? ['voice.sqlite'] : [])]; const manifest = { contract: 's15-composition/v1', installationId: options.installationId, stores: storeNames }; if (options.create) { // mkdir without recursive prevents accidentally accepting an existing S14/legacy/partial directory. @@ -188,7 +195,7 @@ export async function composeS15(options: S15CompositionOptions) { // earlier manifest, which an earlier release needs to start) and may hold an armed effect, or a complete fresh // store (creation is atomic). Only a store this boot created is removed if the boot fails. A store the manifest // already lists must still exist; a missing listed ledger always refuses. - const additive = [...(options.browser?.approved ? ['browser.sqlite'] : []), ...(webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : [])]; + const additive = [...(options.browser?.approved ? ['browser.sqlite'] : []), ...(webGithubEnabled ? ['github.sqlite'] : []), ...(options.files ? ['files.sqlite'] : []), ...(options.routines ? ['routines.sqlite'] : []), ...(options.notifications ? ['notifications.sqlite'] : []), ...(options.bots ? ['bots.sqlite'] : []), ...(options.connections ? ['connections.sqlite'] : []), ...(options.skills ? ['skills.sqlite'] : []), ...(options.voice ? ['voice.sqlite'] : [])]; let added: string[] = []; const created: string[] = []; if (!options.create) { @@ -836,7 +843,13 @@ export async function composeS15(options: S15CompositionOptions) { return (['owner_named_send', 'github_default'] as const).some(mode => proposed.rules.some(r => r.mode === mode) && !current.rules.some(r => r.mode === mode)); } catch { return false; } }; - const clientHandler = createS16ClientHandler(conversation, { origin: options.origin, cookieName, authentication, memory, policyUpdate, models: createS16ModelsPort(models, options.model.registry, policy), ...(routines ? { routines } : {}), ...((routines || computerContinuations) ? { wake: () => { void composed.drain().catch(() => {}); } } : {}), ...(notifications ? { notifications } : {}), ...(computers ? { computers } : {}), ...(screen ? { screen } : {}), ...(browserCards ? { browserCards } : {}), ...(needsYou ? { needsYou } : {}), ...(bots ? { bots } : {}), ...(connections ? { connections } : {}), ...(skills ? { skills } : {}), ...(files ? { files, imageAvailable: () => { const route = options.model.registry.current(options.conversation.selection); return !!route && route.availability === "available" && !!options.files?.imageProbes?.some(probe => probe.verdict === "pass" && probe.route === route.provider && probe.model === route.modelId); } } : {}) }); + const voice = options.voice ? await VoiceService.open({ path: join(root, 'voice.sqlite'), installationId: options.installationId, create: options.create || created.includes('voice.sqlite'), limits: options.voice.quota, now: clockNow, + requireOwner, ownerGeneration: options.owner.generation, + conversation: (owner, id) => { conversationOwnerView(conversation, owner, id); }, + reply: (owner, id, requestId) => { const row = conversationRequestView(conversation, owner, requestId); if (row.receipt.conversationId !== id || !row.result) throw new AppError('voice_result_unavailable', 409); return { revision: row.revision, text: row.result.text }; }, + engine: options.voice.engine ?? (() => openLocalVoice(options.voice!.local, options.voice!.scratchDirectory ?? join(realpathSync(tmpdir()), 'jit-voice-' + new Bun.CryptoHasher('sha256').update(options.installationId + '\n' + realpathSync(root)).digest('hex').slice(0, 24)))) }) : undefined; + if (voice) cleanup.push(() => voice.close()); + const clientHandler = createS16ClientHandler(conversation, { origin: options.origin, cookieName, authentication, memory, policyUpdate, ...(voice ? { voice } : {}), models: createS16ModelsPort(models, options.model.registry, policy), ...(routines ? { routines } : {}), ...((routines || computerContinuations) ? { wake: () => { void composed.drain().catch(() => {}); } } : {}), ...(notifications ? { notifications } : {}), ...(computers ? { computers } : {}), ...(screen ? { screen } : {}), ...(browserCards ? { browserCards } : {}), ...(needsYou ? { needsYou } : {}), ...(bots ? { bots } : {}), ...(connections ? { connections } : {}), ...(skills ? { skills } : {}), ...(files ? { files, imageAvailable: () => { const route = options.model.registry.current(options.conversation.selection); return !!route && route.availability === "available" && !!options.files?.imageProbes?.some(probe => probe.verdict === "pass" && probe.route === route.provider && probe.model === route.modelId); } } : {}) }); const s16Web = await createS16WebHandler(clientHandler); const vmHandler = createVmApi(enrollment, async request => authenticate(request), { service: vm, ...(managers instanceof VmManagers ? { managers } : {}) }); const modelHandler = createModelProjectionHandler(models, authenticate), workHandler = createResponsibilityProjectionHandler(work, authenticate); @@ -854,7 +867,7 @@ export async function composeS15(options: S15CompositionOptions) { const dir = openSync(root, 'r'); try { fsyncSync(dir); } finally { closeSync(dir); } added = []; } - const composed = { policy, conversation, team, memory, files, bots, connections, skills, sources: sourceRuntime, disclosure, adoptionStatus, githubConnections, vm, workingVm, vmStore, enrollment, vmManagers: managers, vmPool: pool, provisioning, computers, computerScopes: scopes, screen, models, work, github, browser, authentication, resumes, routines, computerContinuations, + const composed = { policy, conversation, team, memory, files, bots, connections, skills, voice, sources: sourceRuntime, disclosure, adoptionStatus, githubConnections, vm, workingVm, vmStore, enrollment, vmManagers: managers, vmPool: pool, provisioning, computers, computerScopes: scopes, screen, models, work, github, browser, authentication, resumes, routines, computerContinuations, /** One protected scheduler wake; waiting records alone hold no VM/native session. Stall-reply resumes * settle after CHAT. */ drain(): Promise { if (closing) return Promise.resolve(); try { computerContinuations?.pump(); } catch (error) { return Promise.reject(error); } return draining ??= (async () => { try { reconcileVm(); try { routines?.tick(); routines?.dispatch(); } catch { /* Committed occurrences retry next wake. */ } await conversation.drain(); computerContinuations?.pump(); await conversation.drain(); computerContinuations?.pump(); try { routines?.observe(); } catch { /* Observed next wake. */ } try { resumes.settle(); } catch { /* Pending bindings stay pending. */ } await github?.drain(); } finally { draining = null; } })(); }, diff --git a/src/s15/main.ts b/src/s15/main.ts index 446402d3..ab0ea07c 100644 --- a/src/s15/main.ts +++ b/src/s15/main.ts @@ -133,6 +133,7 @@ export async function runS15(args: string[], env: NodeJS.ProcessEnv = process.en // CONN16-01/MCP16-01/OPEN16-01: apps and tool sources the owner connects; secrets stay in the protected vault. ...(config.connections ? { connections: { vault } } : {}), ...(config.skills ? { skills: {} } : {}), + ...(config.voice ? { voice: config.voice } : {}), ...(config.files ? { files: { ...(typeof config.files === 'object' ? { parserRuntime: binaryFileRuntime(config.files, config.directory, env), imageProbes: config.files.imageProbes } : {}), ...(config.fileMemory ? { memory: true } : {}) } } : {}), recipients, owner: { authenticate: token => { const auth = ownerApp.authenticate(token); return { proof: auth.actor, csrf: auth.csrf }; }, generation, verifyOwner: requireOwner, verifyCsrf: (expected, supplied) => ownerApp.verifyCsrf(expected, supplied) }, catalogs, diff --git a/src/s15/runtime-config.ts b/src/s15/runtime-config.ts index 7c496092..0600dba4 100644 --- a/src/s15/runtime-config.ts +++ b/src/s15/runtime-config.ts @@ -10,6 +10,9 @@ import type { GithubRuntimeConfig } from './github/runtime'; import type { RuntimeDisclosureConfig } from './disclosure/runtime'; import type { S15CompositionOptions } from './composition'; import { validateBinaryFileConfig, type BinaryFileConfig } from '../s16/files/config'; +import { validateLocalVoiceConfig, type LocalVoiceConfig } from '../s16/voice/local'; +import { validateVoiceQuota } from '../s16/voice/service'; +import type { VoiceQuotaConfig } from '../s16/voice/contracts'; /** Credential-free explicit source/disposable-service configuration; never an existing omitted-runtime service file. */ export interface S15RuntimeConfig { @@ -43,6 +46,8 @@ export interface S15RuntimeConfig { connections?: boolean; /** SKILL16-01 pinned package import/review and working-computer use; requires DATA15. */ skills?: boolean; + /** Explicit configured local owner speech. No credential, provider URL, arbitrary flags or automatic installation. */ + voice?: { local: LocalVoiceConfig; quota: VoiceQuotaConfig }; recurring?: import('./policy/recurring').RecurringRuleOptions; /** * The team's working-computer pool (POOL15-01). `id` names the pool and its default member; with `local` the @@ -91,6 +96,10 @@ export function parseS15RuntimeConfig(raw: unknown): S15RuntimeConfig { if (value.bots !== undefined && typeof value.bots !== 'boolean') throw new AppError('invalid_s15_config'); if (value.skills !== undefined && (typeof value.skills !== 'boolean' || value.skills && !value.disclosure)) throw new AppError('invalid_s15_config'); if (value.connections !== undefined && (typeof value.connections !== 'boolean' || value.connections && !value.disclosure)) throw new AppError('invalid_s15_config'); + if (value.voice !== undefined) { + try { if (!value.voice || typeof value.voice !== 'object' || Object.keys(value.voice).sort().join() !== 'local,quota') throw Error(); validateLocalVoiceConfig(value.voice.local); validateVoiceQuota(value.voice.quota); } + catch { throw new AppError('invalid_s15_config'); } + } if (value.policy && (!value.disclosure || value.policy.resources.some(r => !value.disclosure!.sources.some(s => equal(s.resource, r))))) throw new AppError('classification_unknown'); return value; } diff --git a/src/s16/client/client.ts b/src/s16/client/client.ts index bddccb0f..4f37dbbe 100644 --- a/src/s16/client/client.ts +++ b/src/s16/client/client.ts @@ -1,4 +1,6 @@ import type { SkillCommand, SkillCommandStatus, SkillDependencyBinding, SkillFile, SkillReceipt } from "../skills/types"; +import { voiceRequestId, type VoiceCapture, type VoiceCaptureCommand, type VoicePlayback, type VoicePlaybackCommand, type VoiceStatus, type VoiceTranscript } from "../voice/contracts"; +import { captureWavInfo, VoiceAudioError } from "../voice/wav"; import type { SkillDetail, SkillsView } from "../skills/owner-view"; import type { ComputerNeedsYouCard, ComputerNeedsYouCommand, ComputerNeedsYouResult } from "../computers/needs-you-types"; import type { ClientModelsView } from "./contracts"; @@ -63,6 +65,61 @@ export class JitClient { if (!count(data.budget.unsettled)) return invalid(); return data as unknown as ClientModelsView; } + /** Read configured local speech health/quota without microphone, model or process activity. */ + async voiceStatus(signal?: AbortSignal): Promise { + const data = await this.call('/voice', 'GET', undefined, signal); + if (!record(data) || data.contract !== 's16-voice/v1' || typeof data.available !== 'boolean' || data.route !== 'local' || !count(data.serverTime) || !record(data.limits) || !count(data.limits.captureMs) || !count(data.limits.dailyMs) || !this.voiceUsage(data.usage)) return invalid(); + return data as unknown as VoiceStatus; + } + /** Freeze one intentional Mic identity using the service's clock sample, retained through uncertain responses. */ + prepareDictation(serverTime: number): VoiceCaptureCommand { return Object.freeze({ contract: 's16-voice-capture/v1', requestId: voiceRequestId(serverTime) }); } + /** Explicit reserve only; the caller requests microphone permission separately after seeing this receipt. */ + async beginDictation(conversationId: string, command: VoiceCaptureCommand, signal?: AbortSignal): Promise { + const input = structuredClone(command); id(conversationId); id(input.requestId); + const data = await this.call(`/conversations/${conversationId}/voice/captures`, 'POST', input, signal, input.requestId); + if (!this.voiceCapture(data, input.requestId, conversationId)) return invalid(); return data as unknown as VoiceCapture; + } + /** Inspect an original draft; no microphone capture or transcription is reconstructed. */ + async dictationStatus(capture: Pick, signal?: AbortSignal): Promise { + const input = structuredClone(capture); id(input.id); id(input.conversationId); + const data = await this.call(`/voice/${input.id}`, 'GET', undefined, signal); + if (!this.voiceCapture(data, input.id, input.conversationId)) return invalid(); return data as unknown as VoiceCapture; + } + /** Submit strict snapshotted PCM once. The transcript remains editable data; this never calls Send or Decide. */ + async transcribeDictation(capture: Pick, bytes: Uint8Array, signal?: AbortSignal): Promise { + const input = structuredClone(capture), wav = new Uint8Array(bytes); id(input.id); id(input.conversationId); + try { captureWavInfo(wav); } catch (error) { throw new JitClientError(error instanceof VoiceAudioError ? error.code : 'voice_audio_invalid', null, 'not_sent', input.id); } + const data = await this.call(`/voice/${input.id}/transcribe`, 'POST', undefined, signal, input.id, wav); + if (!record(data) || data.contract !== 's16-voice-transcript/v1' || !this.voiceCapture(data.capture, input.id, input.conversationId) || data.capture.state !== 'ready' || typeof data.text !== 'string' || new TextEncoder().encode(data.text).length > 16384 || !this.voiceUsage(data.usage)) return invalid(); + return data as unknown as VoiceTranscript; + } + /** Stop only voice; underlying conversation work and exact approval state remain unchanged. */ + async cancelVoice(operation: { id: string; conversationId: string }, signal?: AbortSignal): Promise { + const input = structuredClone(operation); id(input.id); id(input.conversationId); + const data = await this.call(`/voice/${input.id}/cancel`, 'POST', {}, signal, input.id); + if (!this.voiceCapture(data, input.id, input.conversationId) || data.state !== 'cancelled') return invalid(); return data as unknown as VoiceCapture; + } + /** Freeze a specific currently displayed reply; hashing is local and does not start synthesis or work. */ + async preparePlayback(serverTime: number, conversationId: string, requestId: string, revision: number, text: string): Promise { + const generation = this.#sessionGeneration; id(conversationId); id(requestId); + if (!count(revision) || !revision || typeof text !== 'string' || !text.trim() || text.length > 2000 || new TextEncoder().encode(text).length > 8192) throw new JitClientError('voice_text_limit', null, 'not_sent'); + const textSha256 = Array.from(new Uint8Array(await crypto.subtle.digest('SHA-256', new TextEncoder().encode(text))), byte => byte.toString(16).padStart(2, '0')).join(''); + if (generation !== this.#sessionGeneration) throw new JitClientError('client_response_stale', null, 'not_sent'); + return Object.freeze({ contract: 's16-voice-playback/v1', requestId: voiceRequestId(serverTime), conversationId, targetRequestId: requestId, targetRevision: revision, textSha256 }); + } + /** Synthesize once; late/stale replies use the same transport generation fences as ordinary owner projections. */ + async playReply(command: VoicePlaybackCommand, signal?: AbortSignal): Promise { + const input = structuredClone(command); id(input.requestId); id(input.conversationId); id(input.targetRequestId); + const data = await this.call('/voice/playback', 'POST', input, signal, input.requestId); + if (!record(data) || data.contract !== 's16-voice-playback/v1' || data.id !== input.requestId || data.conversationId !== input.conversationId || data.targetRequestId !== input.targetRequestId || !count(data.durationMs) || (data.durationMs as number) > 60000 || typeof data.wavBase64 !== 'string' || data.wavBase64.length > 5592408 || !this.voiceUsage(data.usage)) return invalid(); + try { const decoded = atob(data.wavBase64); if (!decoded.length || decoded.length > 4 * 1024 * 1024 || btoa(decoded) !== data.wavBase64 || decoded.slice(0, 4) !== 'RIFF' || decoded.slice(8, 12) !== 'WAVE') return invalid(); } + catch { return invalid(); } + return data as unknown as VoicePlayback; + } + private voiceUsage(value: unknown): boolean { return record(value) && value.unit === 'local_audio_ms' && typeof value.day === 'string' && /^\d{4}-\d\d-\d\d$/.test(value.day) && [value.limitMs, value.chargedMs, value.reservedMs, value.conservativeMs].every(count); } + private voiceCapture(value: unknown, expectedId: string, conversationId: string): value is VoiceCapture { + return record(value) && value.id === expectedId && value.conversationId === conversationId && ['capturing', 'processing', 'ready', 'cancelled', 'failed', 'unknown', 'expired'].includes(String(value.state)) && count(value.expiresAt) && count(value.maxDurationMs) && (value.maxDurationMs as number) <= 60000; + } /** Creates one local immutable send identity; callers retain this command through response loss. */ prepareMessage(conversationId: string, text: string, files: readonly FileReference[] = []): ClientMessage { id(conversationId); diff --git a/src/s16/client/contracts.ts b/src/s16/client/contracts.ts index 52eac874..3276ee70 100644 --- a/src/s16/client/contracts.ts +++ b/src/s16/client/contracts.ts @@ -34,6 +34,8 @@ export interface ClientCapabilities { skills?: { available: false; code: "skills_unavailable" } | { available: true; contract: "s16-client-skills/v1" }; /** MODEL16-01: configured routes and recorded work usage; reads never start provider work. */ models?: { available: true; contract: "s16-client-models/v1" }; + /** Owner audio API is configured; actual local binary/model health and quota are read from its status endpoint. */ + voice?: { available: false; code: "voice_unavailable" } | { available: true; contract: "s16-voice/v1"; route: "local" }; } /** Authenticated, no-store bootstrap. CSRF stays in the transport, outside persisted client state. */ export interface ClientBootstrap { diff --git a/src/s16/client/index.ts b/src/s16/client/index.ts index 8f36aa4d..aa96a641 100644 --- a/src/s16/client/index.ts +++ b/src/s16/client/index.ts @@ -2,6 +2,7 @@ export { JitClient, JitClientError } from "./client"; export { beginConversation, emptyConversationState, receiveConversation, withholdConversation } from "./state"; export type * from "./contracts"; +export type * from "../voice/contracts"; export type { ConversationClientState } from "./state"; export { applicationServerKey, disableNotifications, enableNotifications, endpointSha256, notificationLocale, notificationStatus, notificationSupport, notificationWorker, sameApplicationServerKey } from "./notifications"; export type { NotificationEnvironment, NotificationStatus, NotificationSupport } from "./notifications"; diff --git a/src/s16/http.ts b/src/s16/http.ts index ef86b986..875fb5c8 100644 --- a/src/s16/http.ts +++ b/src/s16/http.ts @@ -1,4 +1,7 @@ import type { SkillDirectory } from "./skills/service"; +import type { VoiceService } from "./voice/service"; +import type { VoiceCaptureCommand, VoicePlaybackCommand } from "./voice/contracts"; +import { captureAudio, VoiceAudioError } from "./voice/wav"; import { skillDetail, skillFile, skillsView } from "./skills/owner-view"; import { skillLimits } from "./skills/types"; import { AppError } from "../errors"; @@ -30,7 +33,7 @@ import { connectionLimits } from "./connections/types"; export const artifactPreviewPolicy = "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src 'none'; form-action 'none'; base-uri 'none'; frame-ancestors 'self'"; /** Protected shared client adapter. It adds no engine, database, approval authority or automatic retry. */ -export function createS16ClientHandler(app: ConversationApplication, options: { origin: string; cookieName: string; /** AUTH16-01 banner: true while a newer default proposal awaits the owner. */ policyUpdate?: (actor: Actor) => boolean; authentication: ConversationHttpAuthentication; models?: S16ModelsPort; files?: FileStore; imageAvailable?: () => boolean; memory?: S16MemoryPort; routines?: RoutineService; wake?: () => void; notifications?: NotificationService; computers?: ComputerDirectory; /** DESK16-01 watch and take over a working computer's screen (`s16-computer-screen/v1`). */ screen?: ScreenControl; browserCards?: BrowserActionCards; needsYou?: ComputerNeedsYou; /** BOT16-01 persistent Bots (`s16-client-bots/v1`). */ bots?: BotDirectory; /** CONN16-01 connections (`s16-client-connections/v1`). */ connections?: ConnectionDirectory; /** SKILL16-01 inert owner review and pin/receipt recovery. */ skills?: SkillDirectory }): (request: Request) => Promise { +export function createS16ClientHandler(app: ConversationApplication, options: { origin: string; cookieName: string; /** AUTH16-01 banner: true while a newer default proposal awaits the owner. */ policyUpdate?: (actor: Actor) => boolean; authentication: ConversationHttpAuthentication; models?: S16ModelsPort; files?: FileStore; imageAvailable?: () => boolean; memory?: S16MemoryPort; routines?: RoutineService; wake?: () => void; notifications?: NotificationService; computers?: ComputerDirectory; /** DESK16-01 watch and take over a working computer's screen (`s16-computer-screen/v1`). */ screen?: ScreenControl; browserCards?: BrowserActionCards; needsYou?: ComputerNeedsYou; /** BOT16-01 persistent Bots (`s16-client-bots/v1`). */ bots?: BotDirectory; /** CONN16-01 connections (`s16-client-connections/v1`). */ connections?: ConnectionDirectory; /** SKILL16-01 inert owner review and pin/receipt recovery. */ skills?: SkillDirectory; /** Local owner audio; creates no Bot work or decisions. */ voice?: VoiceService }): (request: Request) => Promise { const origin = new URL(options.origin); if (origin.origin !== options.origin || !/^[A-Za-z0-9_-]+$/.test(options.cookieName)) throw new AppError("invalid_configuration"); const json = (value: unknown, status = 200): Response => { @@ -65,6 +68,7 @@ export function createS16ClientHandler(app: ConversationApplication computers: options.computers ? { available: true, contract: "s16-client-computers/v1", idleStopMs: options.computers.options.vm.idleStopAfterMs, files: !!(files && app.options.files), ...(options.screen ? { screen: true } : {}) } : { available: false, code: "computers_unavailable" }, bots: options.bots ? { available: true, contract: "s16-client-bots/v1", delegation: true } : { available: false, code: "bots_unavailable" }, skills: options.skills ? { available: true, contract: "s16-client-skills/v1" } : { available: false, code: "skills_unavailable" }, + voice: options.voice ? { available: true, contract: "s16-voice/v1", route: "local" } : { available: false, code: "voice_unavailable" }, connections: options.connections ? { available: true, contract: "s16-client-connections/v1", kinds: ["mcp", "openapi", "graphql"], github: !!options.connections.options.github, oauth: true } : { available: false, code: "connections_unavailable" } }; }; return async request => { @@ -85,6 +89,32 @@ export function createS16ClientHandler(app: ConversationApplication if (request.method === "POST") options.authentication.verifyCsrf(current.csrf, request.headers.get("x-csrf-token")); return current.actor; }; + const voicePath = /^\/api\/s16\/client\/voice(?:\/(playback|[A-Za-z0-9_-]{1,128})(?:\/(transcribe|cancel))?)?$/.exec(url.pathname); + const capturePath = /^\/api\/s16\/client\/conversations\/([A-Za-z0-9_-]{1,128})\/voice\/captures$/.exec(url.pathname); + if (voicePath || capturePath) { + if (!options.voice) throw new AppError('voice_unavailable', 503); + if (url.search) throw new AppError('invalid_body'); + if (capturePath && request.method === 'POST') { const capture = await options.voice.begin(currentOwner, capturePath[1]!, await body(request) as VoiceCaptureCommand); return json(options.voice.inspect(currentOwner, capture.id), 201); } + if (voicePath && request.method === 'GET' && !voicePath[1]) { const status = await options.voice.status(currentOwner); currentOwner(); return json(status); } + if (voicePath?.[1] === 'playback' && !voicePath[2] && request.method === 'POST') { const input = await body(request) as VoicePlaybackCommand; await options.voice.playback(currentOwner, input, request.signal); return options.voice.publishPlayback(currentOwner, input, value => json(value)); } + if (voicePath?.[1] && voicePath[1] !== 'playback') { + const captureId = voicePath[1]; + if (request.method === 'GET' && !voicePath[2]) return json(options.voice.inspect(currentOwner, captureId)); + if (request.method === 'POST' && voicePath[2] === 'cancel') { exact(await body(request), []); const cancelled = await options.voice.cancel(currentOwner, captureId); currentOwner(); return json(cancelled); } + if (request.method === 'POST' && voicePath[2] === 'transcribe') { + if (request.headers.get('content-type') !== 'application/octet-stream' || !request.body) throw new AppError('unsupported_content_type', 415); + const capture = options.voice.inspect(currentOwner, captureId); if (capture.state === 'expired') throw new AppError('voice_capture_expired', 409); + const reader = request.body.getReader(), chunks: Uint8Array[] = []; let length = 0, wav: Buffer; + try { + for (;;) { const chunk = await reader.read(); currentOwner(); if (chunk.done) break; length += chunk.value.byteLength; if (length > captureAudio.bytes) throw new AppError('body_too_large', 413); chunks.push(chunk.value); } + currentOwner(); wav = Buffer.concat(chunks); + } finally { await reader.cancel().catch(() => {}); reader.releaseLock(); } + currentOwner(); await options.voice.transcribe(currentOwner, captureId, wav!, request.signal); + return options.voice.publishTranscript(currentOwner, captureId, value => json(value)); + } + } + throw new AppError('method_rejected', 405); + } const uploadCreate = /^\/api\/s16\/client\/conversations\/([A-Za-z0-9_-]{1,128})\/uploads$/.exec(url.pathname); if (uploadCreate && request.method === "POST") { if (!options.files || !app.options.files) throw new AppError("upload_unavailable", 503); @@ -329,6 +359,6 @@ export function createS16ClientHandler(app: ConversationApplication app.stop(current.actor, work[1]!); return json({ state: "stop_requested" }); } throw new AppError("not_found", 404); - } catch (error) { return json({ error: error instanceof AppError ? error.code : "client_unavailable" }, error instanceof AppError ? error.status : 503); } + } catch (error) { return json({ error: error instanceof AppError || error instanceof VoiceAudioError ? error.code : "client_unavailable" }, error instanceof AppError ? error.status : error instanceof VoiceAudioError ? 400 : 503); } }; } diff --git a/src/s16/voice/service.ts b/src/s16/voice/service.ts index edf1a89f..e318f02d 100644 --- a/src/s16/voice/service.ts +++ b/src/s16/voice/service.ts @@ -126,6 +126,20 @@ export class VoiceService { /** Inspect current capture metadata without exposing its transcript/audio or admitting new work. */ inspect(owner: VoiceOwner, id: string): VoiceCapture { this.owner(owner); const record = this.get(id); this.owner(owner, record); return { ...this.capture(record), state: record.state === 'capturing' && record.expiresAt <= this.now() ? 'expired' : record.state }; } + /** Final synchronous draft release, after every transport wait. A transcript is never an approval or message. */ + publishTranscript(owner: VoiceOwner, id: string, publish: (value: VoiceTranscript) => T): T { + this.owner(owner); if (this.get(id).kind !== 'capture') fail('voice_request_invalid'); + return publish(this.cached(id, owner)); + } + /** Reacquire current reply/source permission in the same synchronous turn that the owner transport serializes audio. */ + publishPlayback(owner: VoiceOwner, command: VoicePlaybackCommand, publish: (value: VoicePlayback) => T): T { + this.owner(owner); const record = this.get(command.requestId); this.owner(owner, record); + if (record.kind !== 'playback' || record.commandHash !== sha(canonical(command))) fail('request_id_conflict'); + const reply = this.options.reply(this.owner(owner), command.conversationId, command.targetRequestId); + if (reply.revision !== command.targetRevision || sha(reply.text) !== command.textSha256) fail('voice_result_changed'); + return publish(this.cached(record.id, owner)); + } + private run(owner: VoiceOwner, record: Operation, signal: AbortSignal, body: (context: VoiceProcessContext) => Promise, finalGuard?: () => void): Promise { const abort = new AbortController(), stop = () => abort.abort(); signal.addEventListener('abort', stop, { once: true }); if (signal.aborted) stop(); const guard = () => { if (abort.signal.aborted) fail('voice_cancelled'); const current = this.get(record.id); this.owner(owner, current); if (current.state !== 'processing') fail('voice_cancelled'); }; diff --git a/tests/s16-voice-client.test.ts b/tests/s16-voice-client.test.ts new file mode 100644 index 00000000..0635cb44 --- /dev/null +++ b/tests/s16-voice-client.test.ts @@ -0,0 +1,90 @@ +import { afterEach, expect, test } from 'bun:test'; +import { existsSync } from 'node:fs'; +import { join } from 'node:path'; +import { JitClient } from '../src/s16/client'; +import type { LocalVoiceConfig, LocalVoiceEngine } from '../src/s16/voice/local'; +import { encodeCaptureWav } from '../src/s16/voice/wav'; +import { composedFixture } from './fixtures/s15-composition'; +import { mail } from './fixtures/s15-effects'; + +type Fixture = Awaited>; +const fixtures: Fixture[] = []; +afterEach(async () => { for (const f of fixtures.splice(0).reverse()) await f.close(); }); +const origin = 'http://localhost:3099', audio = () => encodeCaptureWav(new Float32Array(1600)); +const artifact = { path: '/fixture/local-speech', sha256: 'f'.repeat(64) }; +const local: LocalVoiceConfig = { transcription: { executable: artifact, model: artifact, language: 'auto' }, synthesis: { kind: 'espeak', executable: artifact, voice: 'en' }, timeoutMs: 5000 }; +async function fixture(enabled = true) { + let transcriptions = 0, syntheses = 0; + const engine: LocalVoiceEngine = { async available() { return true; }, async transcribe(_input, context) { context.guard(); context.started(); transcriptions++; return 'Approve this draft only.'; }, async synthesize(_text, context) { context.guard(); context.started(); syntheses++; return { wav: audio(), durationMs: 100 }; }, async close() {} }; + const f = await composedFixture('default', false, undefined, options => ({ ...options, ...(enabled ? { voice: { local, quota: { dailyMs: 120000, captureMs: 30000 }, engine: async () => engine } } : {}) })); fixtures.push(f); + const fetch = async (request: Request) => { + const path = new URL(request.url).pathname, headers = Object.fromEntries(request.headers); + if (request.method === 'POST' && request.headers.get('content-type') === 'application/octet-stream') return f.raw(path, await request.arrayBuffer(), headers); + return f.request(path, request.method, request.method === 'POST' ? await request.json() : undefined, headers); + }; + const client = new JitClient({ origin, fetch }); await client.bootstrap(); + return { f, client, fetch, get transcriptions() { return transcriptions; }, get syntheses() { return syntheses; } }; +} + +test('the actual composed client keeps a transcript editable and outside CHAT/model history until ordinary Send', async () => { + const h = await fixture(), { f, client } = h, before = f.service.conversation.store.db.query<{ n: number }, []>('SELECT count(*) AS n FROM requests').get()!.n; + expect((await client.bootstrap()).capabilities.voice).toMatchObject({ available: true, route: 'local' }); + const status = await client.voiceStatus(), capture = await client.beginDictation('conversation', client.prepareDictation(status.serverTime)); + const transcript = await client.transcribeDictation(capture, audio()); + expect(transcript.text).toBe('Approve this draft only.'); expect(h.transcriptions).toBe(1); expect(f.modelPrompts).toEqual([]); expect(f.interpreterPrompts).toEqual([]); + expect(f.service.conversation.store.db.query<{ n: number }, []>('SELECT count(*) AS n FROM requests').get()!.n).toBe(before); + await client.cancelVoice(capture); expect((await client.voiceStatus()).usage).toMatchObject({ chargedMs: 100, reservedMs: 0 }); + expect(f.service.conversation.store.db.query<{ n: number }, []>('SELECT count(*) AS n FROM requests').get()!.n).toBe(before); + const command = client.prepareMessage('conversation', transcript.text + ' Edited before sending.'); await client.send(command); await f.service.conversation.drain(); + const row = (await client.snapshot('conversation')).conversation.rows[0]!; expect(row.ownerMessage).toBe(command.text); expect(row.result).not.toBeNull(); + const playback = client.playReply(await client.preparePlayback((await client.voiceStatus()).serverTime, 'conversation', row.requestId, row.revision, row.result!.text)); + expect((await playback).wavBase64).toBe(Buffer.from(audio()).toString('base64')); expect(h.syntheses).toBe(1); +}); + +test('transcribed approval words use ordinary text ingress and cannot approve an existing held action', async () => { + const h = await fixture(), { f, client } = h; + f.interpretation = { intent: 'execute', purposeId: 'purpose', resourceIds: ['account'], access: 'effect', subject: null }; + f.script = [{ type: 'tool_call', id: 'email', name: 'gmail_prepare_send', arguments: { ...mail() } }]; + const original = client.prepareMessage('conversation', 'Send the prepared email.'); await client.send(original); await f.service.conversation.drain(); + expect((await client.request(original.requestId)).row.slips[0]?.state).toBe('pending'); + const capture = await client.beginDictation('conversation', client.prepareDictation((await client.voiceStatus()).serverTime)), transcript = await client.transcribeDictation(capture, audio()); + f.interpretation = { intent: 'inspect', purposeId: 'purpose', resourceIds: ['computer'], access: 'read', subject: null }; f.script = [{ type: 'text', text: 'The transcript is ordinary text.' }]; + await client.send(client.prepareMessage('conversation', transcript.text)); await f.service.conversation.drain(); + expect(f.google.sent.size).toBe(0); expect((await client.request(original.requestId)).row.slips[0]?.state).toBe('pending'); +}); + +test('voice raw-body ingress reauthenticates after streaming and refuses oversized or malformed WAV before processing', async () => { + const h = await fixture(), { f, client } = h, capture = await client.beginDictation('conversation', client.prepareDictation((await client.voiceStatus()).serverTime)); + expect((await f.raw(`/api/s16/client/voice/${capture.id}/transcribe`, new Uint8Array(960045))).status).toBe(413); + expect(await (await f.raw(`/api/s16/client/voice/${capture.id}/transcribe`, new Uint8Array(64))).json()).toEqual({ error: 'voice_audio_invalid' }); + const bytes = audio(), reading = Promise.withResolvers(), release = Promise.withResolvers(); let pull = 0; + const stream = new ReadableStream({ async pull(controller) { if (pull++ === 0) controller.enqueue(bytes.subarray(0, 44)); else { reading.resolve(); await release.promise; controller.enqueue(bytes.subarray(44)); controller.close(); } } }); + const pending = f.raw(`/api/s16/client/voice/${capture.id}/transcribe`, stream); await reading.promise; f.generation = 2; release.resolve(); const response = await pending; + expect(await response.json()).toEqual({ error: 'voice_stale' }); expect(h.transcriptions).toBe(0); +}); + +test('withdrawal after synthesis resolves but before HTTP release withholds the cached audio at the final publisher', async () => { + const { f, client } = await fixture(); await client.send(client.prepareMessage('conversation', 'Run a private diagnostic')); await f.service.conversation.drain(); + const row = (await client.snapshot('conversation')).conversation.rows[0]!, voice = f.service.voice!, original = voice.playback.bind(voice); + voice.playback = async (...args) => { const result = await original(...args); f.dataSources[0]!.active = false; return result; }; + const command = await client.preparePlayback((await client.voiceStatus()).serverTime, 'conversation', row.requestId, row.revision, row.result!.text); + await expect(client.playReply(command)).rejects.toMatchObject({ code: 'voice_result_unavailable', status: 409 }); + expect((await client.request(row.requestId)).row.result).toBeNull(); +}); + +test('late voice JSON after client clear cannot restore private draft text or the prior session', async () => { + const h = await fixture(), { f, client: first } = h, capture = await first.beginDictation('conversation', first.prepareDictation((await first.voiceStatus()).serverTime)); + const entered = Promise.withResolvers(), release = Promise.withResolvers(); + const client = new JitClient({ origin, fetch: async request => { const response = await h.fetch(request); if (new URL(request.url).pathname.endsWith('/transcribe')) { entered.resolve(); await release.promise; } return response; } }); + await client.bootstrap(); const pending = client.transcribeDictation(capture, audio()); await entered.promise; client.clear(); release.resolve(); + await expect(pending).rejects.toMatchObject({ code: 'client_response_stale', outcome: 'unknown', requestId: capture.id }); expect(h.transcriptions).toBe(1); + expect(f.service.conversation.store.db.query<{ n: number }, []>('SELECT count(*) AS n FROM requests').get()!.n).toBe(0); +}); + +test('omitting voice preserves text service and creates no voice journal', async () => { + const { f, client } = await fixture(false); + expect((await client.bootstrap()).capabilities.voice).toMatchObject({ available: false }); expect(existsSync(join(f.directory, 'voice.sqlite'))).toBe(false); + await expect(client.voiceStatus()).rejects.toMatchObject({ code: 'voice_unavailable', status: 503 }); + await client.send(client.prepareMessage('conversation', 'Continue through ordinary text.')); await f.service.conversation.drain(); + expect((await client.snapshot('conversation')).conversation.rows[0]!.result).not.toBeNull(); +}); From e589e4e0831d5265f96b345d773362790065dbe8 Mon Sep 17 00:00:00 2001 From: Chayapat Sriwattanachote Date: Wed, 18 Nov 2026 12:01:16 +0700 Subject: [PATCH 06/11] Add bounded microphone capture and editable voice controls --- src/i18n/en.ts | 23 ++++++ src/i18n/th.ts | 23 ++++++ src/s16/client/voice-recording.ts | 69 ++++++++++++++++ src/s16/http.ts | 2 +- src/s16/voice/local.ts | 13 ++- src/s16/voice/service.ts | 4 +- src/s16/web/VoiceControls.tsx | 98 ++++++++++++++++++++++ src/s16/web/app.tsx | 17 ++++ src/s16/web/handler.ts | 4 +- src/s16/web/style.css | 3 + src/s16/web/voice-worklet.js | 34 ++++++++ tests/s16-voice-audio.test.ts | 8 ++ tests/s16-voice-service.test.ts | 7 ++ tests/s16-voice.browser.test.ts | 133 ++++++++++++++++++++++++++++++ tests/s16-voice.native.test.ts | 60 ++++++++++++++ 15 files changed, 491 insertions(+), 7 deletions(-) create mode 100644 src/s16/client/voice-recording.ts create mode 100644 src/s16/web/VoiceControls.tsx create mode 100644 src/s16/web/voice-worklet.js create mode 100644 tests/s16-voice.browser.test.ts create mode 100644 tests/s16-voice.native.test.ts diff --git a/src/i18n/en.ts b/src/i18n/en.ts index 7f556616..496f2e57 100644 --- a/src/i18n/en.ts +++ b/src/i18n/en.ts @@ -6504,4 +6504,27 @@ export const en: Record = { "s16.connections.card.note": "Nothing is connected or approved until you finish on the protected form. Never paste secrets into chat.", "s16.connections.card.open": "Open the connection form", "s16.connections.card.dismiss": "Dismiss", + "s16.voice.title": "Voice", + "s16.voice.dictate": "Dictate", + "s16.voice.stop": "Stop recording", + "s16.voice.cancel": "Discard dictation", + "s16.voice.interrupt": "Stop listening", + "s16.voice.preparing": "Preparing voice…", + "s16.voice.permission": "Waiting for microphone permission…", + "s16.voice.recording": "Recording", + "s16.voice.transcribing": "Transcribing…", + "s16.voice.loading": "Preparing audio…", + "s16.voice.playing": "Listening to reply", + "s16.voice.checking": "Checking local voice…", + "s16.voice.unavailable": "Local voice is unavailable. You can keep using text.", + "s16.voice.local": "Audio is processed locally on your Jit server.", + "s16.voice.usage": "{used} s processed · {reserved} s reserved · {limit} s daily limit", + "s16.voice.conservative": "{seconds} s remain conservatively charged after an interrupted operation.", + "s16.voice.transcript": "Edit your transcript", + "s16.voice.review": "Review this text before adding it to your message. Speech never approves an action.", + "s16.voice.useText": "Add to message", + "s16.voice.listen": "Listen to this reply", + "s16.voice.permissionDenied": "Microphone access stopped or was denied. Your text is still available.", + "s16.voice.budget": "The local voice limit is reached. Continue with text.", + "s16.voice.failed": "Voice stopped. You can continue with text.", }; diff --git a/src/i18n/th.ts b/src/i18n/th.ts index 2718a6af..6e793322 100644 --- a/src/i18n/th.ts +++ b/src/i18n/th.ts @@ -6508,4 +6508,27 @@ export const th = { "s16.connections.card.note": "ยังไม่มีอะไรถูกเชื่อมหรืออนุมัติจนกว่าคุณทำให้เสร็จในฟอร์มที่ปลอดภัย อย่าวางข้อมูลลับในแชท", "s16.connections.card.open": "เปิดฟอร์มเชื่อมต่อ", "s16.connections.card.dismiss": "ปิด", + "s16.voice.title": "เสียง", + "s16.voice.dictate": "พิมพ์ด้วยเสียง", + "s16.voice.stop": "หยุดอัดเสียง", + "s16.voice.cancel": "ทิ้งข้อความเสียง", + "s16.voice.interrupt": "หยุดฟัง", + "s16.voice.preparing": "กำลังเตรียมเสียง…", + "s16.voice.permission": "กำลังรอสิทธิ์ใช้ไมโครโฟน…", + "s16.voice.recording": "กำลังอัดเสียง", + "s16.voice.transcribing": "กำลังถอดเสียง…", + "s16.voice.loading": "กำลังเตรียมเสียงอ่าน…", + "s16.voice.playing": "กำลังฟังคำตอบ", + "s16.voice.checking": "กำลังตรวจเสียงบนเครื่อง…", + "s16.voice.unavailable": "เสียงบนเครื่องยังใช้ไม่ได้ คุณยังใช้ข้อความต่อได้", + "s16.voice.local": "ประมวลผลเสียงบนเซิร์ฟเวอร์ Jit ของคุณ", + "s16.voice.usage": "ประมวลผลแล้ว {used} วิ · สำรอง {reserved} วิ · จำกัดวันละ {limit} วิ", + "s16.voice.conservative": "ยังนับยอดสำรอง {seconds} วิ จากรายการที่ถูกขัดจังหวะ", + "s16.voice.transcript": "แก้ไขข้อความที่ถอดเสียง", + "s16.voice.review": "ตรวจข้อความก่อนเพิ่มในข้อความที่จะส่ง เสียงพูดไม่อนุมัติการกระทำใด ๆ", + "s16.voice.useText": "เพิ่มในข้อความ", + "s16.voice.listen": "ฟังคำตอบนี้", + "s16.voice.permissionDenied": "สิทธิ์ไมโครโฟนถูกปฏิเสธหรือหยุดแล้ว คุณยังใช้ข้อความได้", + "s16.voice.budget": "ใช้เสียงถึงขีดจำกัดแล้ว ใช้ข้อความต่อได้", + "s16.voice.failed": "เสียงหยุดแล้ว คุณยังใช้ข้อความต่อได้", } as const; diff --git a/src/s16/client/voice-recording.ts b/src/s16/client/voice-recording.ts new file mode 100644 index 00000000..b6ec2c07 --- /dev/null +++ b/src/s16/client/voice-recording.ts @@ -0,0 +1,69 @@ +import { encodeCaptureWav } from '../voice/wav'; + +/** Fixed authenticated same-origin AudioWorklet module; callers never supply scripts or a remote recognizer. */ +export const voiceWorkletPath = '/s16/voice-worklet.js'; +/** One finite local microphone lease. Stop produces a draft clip; cancel discards it and never submits text. */ +export interface VoiceRecording { done: Promise; stop(): void; cancel(): Promise } +const failure = (code: string) => new Error(code); +const stopTracks = (stream?: MediaStream) => { for (const track of stream?.getTracks() ?? []) track.stop(); }; + +/** Explicit gesture-only capture with late-permission fencing, finite PCM buffers and native track/node cleanup. */ +export async function startVoiceRecording(options: { signal: AbortSignal; durationMs: number; deadline?: number; progress(ms: number): void }): Promise { + if (!navigator.mediaDevices?.getUserMedia || typeof AudioContext !== 'function' || typeof AudioWorkletNode !== 'function') throw failure('voice_capture_unsupported'); + if (!Number.isFinite(options.durationMs) || options.durationMs < 1 || options.durationMs > 30000) throw failure('voice_capture_expired'); + if (options.signal.aborted) throw failure('voice_cancelled'); + let stream: MediaStream | undefined, context: AudioContext | undefined, source: MediaStreamAudioSourceNode | undefined, node: AudioWorkletNode | undefined, silent: GainNode | undefined; + const deadline = options.deadline ?? performance.now() + options.durationMs + 15000; + let stopping = false, finished = false, frames = 0, maximum = Math.floor(options.durationMs * 16), chunks: Float32Array[] = []; + let timeout: ReturnType | undefined, watchdog: ReturnType | undefined; + const complete = Promise.withResolvers(); void complete.promise.catch(() => {}); + const interrupted = Promise.withResolvers(); void interrupted.promise.catch(() => {}); + let closing: Promise | null = null; + const close = (): Promise => { + stopping = true; stopTracks(stream); clearTimeout(timeout); clearTimeout(watchdog); + if (node) { node.port.onmessage = null; node.port.close(); node.disconnect(); } source?.disconnect(); silent?.disconnect(); + return closing ??= (async () => { if (context && context.state !== 'closed') await context.close().catch(() => {}); })(); + }; + const refuse = async (code: string) => { interrupted.reject(failure(code)); if (finished) return; finished = true; chunks = []; await close(); complete.reject(failure(code)); }; + const abort = () => { void refuse('voice_cancelled'); }; + options.signal.addEventListener('abort', abort, { once: true }); + const done = complete.promise.finally(() => options.signal.removeEventListener('abort', abort)); void done.catch(() => {}); + timeout = setTimeout(() => { void refuse('voice_capture_expired'); }, Math.max(0, deadline - performance.now())); + try { + const pending = navigator.mediaDevices.getUserMedia({ audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true }, video: false }); + // A cancelled permission prompt may resolve later. Its stream is stopped even when this caller already returned. + void pending.then(value => { if (options.signal.aborted || finished) stopTracks(value); }, () => {}); + stream = await Promise.race([pending, interrupted.promise]); + if (options.signal.aborted || finished) { stopTracks(stream); throw failure('voice_cancelled'); } + const lost = () => { if (!stopping) void refuse('voice_permission_lost'); }; + for (const track of stream.getAudioTracks()) { track.addEventListener('ended', lost, { once: true }); track.addEventListener('mute', lost, { once: true }); } + context = new AudioContext(); await Promise.race([context.audioWorklet.addModule(voiceWorkletPath, { credentials: 'same-origin' }), interrupted.promise]); + if (options.signal.aborted || finished) throw failure('voice_cancelled'); + maximum = Math.floor(Math.min(options.durationMs, deadline - performance.now()) * 16); if (maximum < 1) throw failure('voice_capture_expired'); + node = new AudioWorkletNode(context, 'jit-voice-capture', { numberOfInputs: 1, numberOfOutputs: 1, outputChannelCount: [1], processorOptions: { maximumFrames: maximum } }); + const finish = async () => { + if (finished) return; finished = true; + try { + if (!frames) throw failure('voice_audio_empty'); + const samples = new Float32Array(frames); let at = 0; for (const chunk of chunks) { samples.set(chunk, at); at += chunk.length; } + chunks = []; const wav = encodeCaptureWav(samples); await close(); + if (options.signal.aborted) throw failure('voice_cancelled'); complete.resolve(wav); + } catch (error) { chunks = []; await close(); complete.reject(error); } + }; + node.port.onmessage = event => { + if (finished || options.signal.aborted) return; + const data = event.data; + if (data?.kind === 'samples' && data.samples instanceof Float32Array && data.samples.length > 0 && data.samples.length <= 1024 && frames + data.samples.length <= maximum) { chunks.push(data.samples); frames += data.samples.length; options.progress(frames / 16); } + else if (data?.kind === 'ended') void finish(); + else void refuse('voice_audio_invalid'); + }; + source = context.createMediaStreamSource(stream); silent = context.createGain(); silent.gain.value = 0; source.connect(node); node.connect(silent); silent.connect(context.destination); + await Promise.race([context.resume(), interrupted.promise]); if (options.signal.aborted) throw failure('voice_cancelled'); + const stop = () => { if (finished || stopping) return; stopping = true; stopTracks(stream); clearTimeout(timeout); node!.port.postMessage({ kind: 'stop' }); watchdog = setTimeout(() => { void refuse('voice_capture_unavailable'); }, 1500); }; + if (!finished) { clearTimeout(timeout); timeout = setTimeout(stop, Math.max(0, Math.min(options.durationMs, deadline - performance.now()))); } + return { done, stop, async cancel() { await refuse('voice_cancelled'); await close(); } }; + } catch (error) { + const name = error instanceof Error ? error.name : '', code = error instanceof Error && /^voice_[a-z_]+$/.test(error.message) ? error.message : name === 'NotAllowedError' ? 'voice_permission_denied' : ['NotFoundError', 'NotReadableError'].includes(name) ? 'voice_microphone_unavailable' : 'voice_capture_unavailable'; + await refuse(code); await close(); throw failure(code); + } +} diff --git a/src/s16/http.ts b/src/s16/http.ts index 875fb5c8..789c858a 100644 --- a/src/s16/http.ts +++ b/src/s16/http.ts @@ -94,7 +94,7 @@ export function createS16ClientHandler(app: ConversationApplication if (voicePath || capturePath) { if (!options.voice) throw new AppError('voice_unavailable', 503); if (url.search) throw new AppError('invalid_body'); - if (capturePath && request.method === 'POST') { const capture = await options.voice.begin(currentOwner, capturePath[1]!, await body(request) as VoiceCaptureCommand); return json(options.voice.inspect(currentOwner, capture.id), 201); } + if (capturePath && request.method === 'POST') { const capture = await options.voice.begin(currentOwner, capturePath[1]!, await body(request) as VoiceCaptureCommand, request.signal); return json(options.voice.inspect(currentOwner, capture.id), 201); } if (voicePath && request.method === 'GET' && !voicePath[1]) { const status = await options.voice.status(currentOwner); currentOwner(); return json(status); } if (voicePath?.[1] === 'playback' && !voicePath[2] && request.method === 'POST') { const input = await body(request) as VoicePlaybackCommand; await options.voice.playback(currentOwner, input, request.signal); return options.voice.publishPlayback(currentOwner, input, value => json(value)); } if (voicePath?.[1] && voicePath[1] !== 'playback') { diff --git a/src/s16/voice/local.ts b/src/s16/voice/local.ts index c8a844a3..fd286c8d 100644 --- a/src/s16/voice/local.ts +++ b/src/s16/voice/local.ts @@ -47,15 +47,18 @@ export function validateLocalVoiceConfig(value: unknown): asserts value is Local || !Number.isSafeInteger(value.timeoutMs) || (value.timeoutMs as number) < 1 || (value.timeoutMs as number) > 120000) throw new AppError('invalid_voice_config'); } -async function verifyArtifact(value: VoiceArtifact, executable: boolean, signal: AbortSignal): Promise { +async function verifyArtifact(value: VoiceArtifact, executable: boolean, signal: AbortSignal, health?: Map): Promise { if (await realpath(value.path) !== value.path) reject('voice_artifact_changed'); const handle = await open(value.path, constants.O_RDONLY | constants.O_NOFOLLOW); try { const stat = await handle.stat(); if (!stat.isFile() || stat.size < 1 || stat.size > (executable ? 256 * 1024 * 1024 : 1024 * 1024 * 1024) || stat.mode & 0o022 || executable && !(stat.mode & 0o111)) reject('voice_artifact_changed'); + const key = value.path + '\n' + value.sha256, fingerprint = [stat.dev, stat.ino, stat.size, stat.mode, stat.mtimeMs, stat.ctimeMs].join(':'); + if (health?.get(key) === fingerprint) return; const hash = createHash('sha256'), buffer = Buffer.alloc(65536); let total = 0; for (;;) { signal.throwIfAborted(); const read = await handle.read(buffer, 0, buffer.length, null); if (!read.bytesRead) break; total += read.bytesRead; if (total > stat.size) reject('voice_artifact_changed'); hash.update(buffer.subarray(0, read.bytesRead)); } if (total !== stat.size || hash.digest('hex') !== value.sha256) reject('voice_artifact_changed'); + health?.set(key, fingerprint); } finally { await handle.close(); } } @@ -76,7 +79,7 @@ export function speechWavInfo(bytes: Uint8Array): { durationMs: number } { if (rate < 8000 || rate > 48000 || alignment !== 2 || view.getUint32(start + 8, true) !== rate * alignment || length === 18 && view.getUint16(start + 16, true) !== 0) return invalid(); format = { rate, alignment }; } else if (kind === 'data') { if (size !== undefined || !length || length % 2) return invalid(); size = length; } - else if (!['JUNK', 'LIST', 'fact'].includes(kind)) return invalid(); + else if (!['JUNK', 'FLLR', 'LIST', 'fact'].includes(kind)) return invalid(); at = end + length % 2; } if (at !== bytes.length || !format || !size) return invalid(); @@ -117,7 +120,8 @@ export async function openLocalVoice(configured: LocalVoiceConfig, scratchPath: recovered = true; return true; })().finally(() => { recovery = null; }); }; - const verify = async (signal: AbortSignal) => { if (config.synthesis.kind === 'say' && process.platform !== 'darwin') reject('voice_engine_unavailable'); await verifyArtifact(config.transcription.executable, true, signal); await verifyArtifact(config.transcription.model, false, signal); await verifyArtifact(config.synthesis.executable, true, signal); }; + const health = new Map(); + const verify = async (signal: AbortSignal, cache?: Map) => { if (config.synthesis.kind === 'say' && process.platform !== 'darwin') reject('voice_engine_unavailable'); await verifyArtifact(config.transcription.executable, true, signal, cache); await verifyArtifact(config.transcription.model, false, signal, cache); await verifyArtifact(config.synthesis.executable, true, signal, cache); }; const job = (context: VoiceProcessContext, body: (directory: string, context: VoiceProcessContext) => Promise): Promise => { const signal = AbortSignal.any([context.signal, shutdown.signal]), check = () => { if (closed || signal.aborted) reject('voice_cancelled'); context.guard(); }; const running = (async () => { @@ -145,7 +149,8 @@ export async function openLocalVoice(configured: LocalVoiceConfig, scratchPath: } finally { context.signal.removeEventListener('abort', stop); stop(); await worker.exited; } }; return { - async available() { if (closed) return false; try { if (!await recover()) return false; await verify(shutdown.signal); return !closed; } catch { return false; } }, + // Health reads can reuse unchanged file metadata. Every actual speech job still rehashes all pinned artifacts. + async available() { if (closed) return false; try { if (!await recover()) return false; await verify(shutdown.signal, health); return !closed; } catch { return false; } }, transcribe(input, context) { const wav = input.slice(); captureWavInfo(wav); return job(context, async (directory, current) => { diff --git a/src/s16/voice/service.ts b/src/s16/voice/service.ts index e318f02d..f448e3e6 100644 --- a/src/s16/voice/service.ts +++ b/src/s16/voice/service.ts @@ -110,12 +110,14 @@ export class VoiceService { const at = this.now(); return { contract: 's16-voice/v1', available, serverTime: at, route: 'local', limits: { ...this.options.limits }, usage: this.usageAt(at) }; } /** Reserve a bounded draft. Retried/expired identities cannot silently create another capture. */ - async begin(owner: VoiceOwner, conversationId: string, input: VoiceCaptureCommand): Promise { + async begin(owner: VoiceOwner, conversationId: string, input: VoiceCaptureCommand, signal?: AbortSignal): Promise { + if (signal?.aborted) fail('voice_cancelled'); exact(input, ['contract', 'requestId']); if (input.contract !== 's16-voice-capture/v1') fail('voice_request_invalid', 400); identity(conversationId); stamp(input.requestId); const command = structuredClone(input), hash = sha(canonical({ ...command, conversationId })); this.options.conversation(this.owner(owner), conversationId); if (!await this.engine.available()) fail('voice_unavailable', 503); return this.journal.tx(() => { + if (signal?.aborted) fail('voice_cancelled'); this.options.conversation(this.owner(owner), conversationId); const at = this.clock(), old = this.find(command.requestId); if (old) { if (old.commandHash !== hash) fail('request_id_conflict'); this.owner(owner, old); this.prune(at); return this.capture(this.get(old.id)); } this.fresh(command.requestId, at); this.capacity(at, this.options.limits.captureMs); diff --git a/src/s16/web/VoiceControls.tsx b/src/s16/web/VoiceControls.tsx new file mode 100644 index 00000000..8f50edff --- /dev/null +++ b/src/s16/web/VoiceControls.tsx @@ -0,0 +1,98 @@ +import { useEffect, useId, useLayoutEffect, useRef, useState } from 'react'; +import { JitClient, JitClientError } from '../client'; +import { startVoiceRecording, type VoiceRecording } from '../client/voice-recording'; +import type { VoiceStatus } from '../voice/contracts'; +import type { Copy } from './copy'; + +type Phase = 'idle' | 'reserving' | 'permission' | 'recording' | 'transcribing' | 'draft' | 'loading' | 'playing' | 'error'; +interface Slot { kind: 'capture' | 'playback'; scope: string; conversationId: string; stop: AbortController; id?: string; recorder?: VoiceRecording; audio?: HTMLAudioElement; url?: string; timer?: ReturnType; target?: { requestId: string; revision: number; text: string } } +const code = (error: unknown) => error instanceof JitClientError ? error.code : error instanceof Error && /^[a-z][a-z0-9_]{0,127}$/.test(error.message) ? error.message : 'voice_unavailable'; + +/** One client voice lifecycle shared by the composer and reply buttons; it never sends a transcript or votes. */ +export function useVoice(options: { client: JitClient; enabled: boolean; canCapture: boolean; conversationId: string; scope: string; onTranscript(text: string): void; onFailure(error: unknown): void }) { + const latest = useRef(options); latest.current = options; + const slot = useRef(null), turn = useRef(0), mounted = useRef(true); + const [phase, setPhase] = useState('idle'), [status, setStatus] = useState(null), [error, setError] = useState(null), [elapsed, setElapsed] = useState(0), [draft, setDraft] = useState(''); + const alive = (value: Slot) => mounted.current && slot.current === value && latest.current.enabled && latest.current.scope === value.scope; + const dispose = async (value: Slot | null, remote = true) => { + if (!value) return; + clearInterval(value.timer); value.stop.abort(); value.audio?.pause(); + if (value.audio) { value.audio.onended = null; value.audio.onerror = null; value.audio.removeAttribute('src'); value.audio.load(); } + if (value.url) URL.revokeObjectURL(value.url); + await Promise.allSettled([value.recorder?.cancel(), remote && value.id ? latest.current.client.cancelVoice({ id: value.id, conversationId: value.conversationId }) : undefined]); + }; + const cancel = () => { turn.current++; const previous = slot.current; slot.current = null; setDraft(''); setElapsed(0); setError(null); setPhase('idle'); void dispose(previous); }; + const failed = async (value: Slot, failure: unknown) => { + if (!alive(value)) return; + slot.current = null; setDraft(''); setError(code(failure)); setPhase('error'); const cleanup = dispose(value); + if (mounted.current && latest.current.scope === value.scope && failure instanceof JitClientError && ([401, 403].includes(failure.status ?? 0) || ['voice_result_changed', 'voice_result_unavailable'].includes(failure.code))) latest.current.onFailure(failure); + await cleanup; + }; + const inspect = async (value: Slot) => { const result = await latest.current.client.voiceStatus(value.stop.signal); if (!alive(value)) throw Error('voice_cancelled'); setStatus(result); if (!result.available) throw Error('voice_unavailable'); return result; }; + const monitor = (value: Slot) => { let checking = false; value.timer = setInterval(() => { if (!alive(value) || checking) return; checking = true; void inspect(value).catch(error => failed(value, error)).finally(() => { checking = false; }); }, 2000); }; + const begin = async (kind: Slot['kind']): Promise => { + const sequence = ++turn.current, scope = latest.current.scope, previous = slot.current; slot.current = null; + setDraft(''); setError(null); setElapsed(0); setPhase(kind === 'capture' ? 'reserving' : 'loading'); await dispose(previous); + if (!mounted.current || turn.current !== sequence || latest.current.scope !== scope || !latest.current.enabled || kind === 'capture' && !latest.current.canCapture) return null; + const value: Slot = { kind, scope, conversationId: latest.current.conversationId, stop: new AbortController() }; slot.current = value; return value; + }; + const record = async () => { + const value = await begin('capture'); if (!value) return; + try { + const requested = performance.now(), status = await inspect(value), command = latest.current.client.prepareDictation(status.serverTime); value.id = command.requestId; + const capture = await latest.current.client.beginDictation(value.conversationId, command, value.stop.signal); + if (!alive(value)) return; if (capture.state !== 'capturing') throw Error('voice_capture_expired'); + setPhase('permission'); monitor(value); + const recorder = await startVoiceRecording({ signal: value.stop.signal, durationMs: capture.maxDurationMs, deadline: requested + capture.expiresAt - status.serverTime, progress(ms) { if (alive(value)) setElapsed(old => Math.floor(old / 1000) === Math.floor(ms / 1000) ? old : ms); } }); + value.recorder = recorder; + if (!alive(value)) { await recorder.cancel(); return; } + setPhase('recording'); const wav = await recorder.done; + delete value.recorder; + if (!alive(value)) return; clearInterval(value.timer); setPhase('transcribing'); + const transcript = await latest.current.client.transcribeDictation(capture, wav, value.stop.signal); + if (!alive(value)) return; setDraft(transcript.text); setPhase('draft'); setStatus(old => old ? { ...old, usage: transcript.usage } : old); + void latest.current.client.voiceStatus(value.stop.signal).then(fresh => { if (alive(value)) setStatus(fresh); }).catch(error => { if (!alive(value)) return; if (error instanceof JitClientError && [401, 403].includes(error.status ?? 0)) void failed(value, error); else setError(code(error)); }); + } catch (error) { await failed(value, error); } + }; + const play = async (target: { requestId: string; revision: number; text: string }) => { + const value = await begin('playback'); if (!value) return; value.target = { ...target }; + try { + const status = await inspect(value), command = await latest.current.client.preparePlayback(status.serverTime, value.conversationId, target.requestId, target.revision, target.text); value.id = command.requestId; + const result = await latest.current.client.playReply(command, value.stop.signal); if (!alive(value)) return; + const bytes = Uint8Array.from(atob(result.wavBase64), character => character.charCodeAt(0)); + value.url = URL.createObjectURL(new Blob([bytes], { type: 'audio/wav' })); value.audio = new Audio(value.url); + value.audio.onended = () => { if (!alive(value)) return; slot.current = null; setPhase('idle'); void dispose(value, false); }; + value.audio.onerror = () => { void failed(value, Error('voice_playback_unavailable')); }; + await value.audio.play(); if (!alive(value)) { value.audio.pause(); return; } + setStatus(old => old ? { ...old, usage: result.usage } : old); setPhase('playing'); + } catch (error) { await failed(value, error); } + }; + const apply = () => { if (!draft.trim() || !slot.current || !alive(slot.current)) return; try { latest.current.onTranscript(draft); cancel(); } catch (error) { setError(code(error)); } }; + useLayoutEffect(() => { + mounted.current = true; let fresh = true; const stop = new AbortController(); + setDraft(''); setError(null); setElapsed(0); setPhase('idle'); setStatus(null); + if (options.enabled) void options.client.voiceStatus(stop.signal).then(status => { if (fresh) setStatus(status); }).catch(error => { if (fresh && !stop.signal.aborted) { setError(code(error)); if (error instanceof JitClientError && [401, 403].includes(error.status ?? 0)) latest.current.onFailure(error); } }); else setStatus(null); + return () => { fresh = false; mounted.current = false; stop.abort(); turn.current++; const previous = slot.current; slot.current = null; void dispose(previous); }; + }, [options.enabled, options.scope, options.client]); + useEffect(() => { if (!options.canCapture && slot.current?.kind === 'capture') cancel(); }, [options.canCapture]); + return { phase, status, error, elapsed, draft, setDraft, record, play, cancel, apply, stop: () => slot.current?.recorder?.stop(), playing: slot.current?.kind === 'playback' ? slot.current.target?.requestId : null, + currentReply: (rows: { requestId: string; revision: number; result: { text: string } | null }[]) => { const current = slot.current; if (current?.kind === 'playback' && current.target && !rows.some(row => row.requestId === current.target!.requestId && row.revision === current.target!.revision && row.result?.text === current.target!.text)) cancel(); } }; +} + +/** Visible recording/transcription/playback states; transcript editing precedes ordinary composer Send. */ +export function VoiceControls({ voice, t, canCapture }: { voice: ReturnType; t: Copy; canCapture: boolean }) { + const transcript = useRef(null), transcriptId = useId(); + useEffect(() => { if (voice.phase === 'draft') transcript.current?.focus(); }, [voice.phase]); + const busy = !['idle', 'error', 'draft'].includes(voice.phase), seconds = Math.floor(voice.elapsed / 1000), usage = voice.status?.usage; + return
+
+ {voice.phase === 'recording' && <>{String(Math.floor(seconds / 60)).padStart(2, '0')}:{String(seconds % 60).padStart(2, '0')}} + {(busy || voice.phase === 'draft') && } + {busy && {t(voice.phase === 'reserving' ? 's16.voice.preparing' : voice.phase === 'permission' ? 's16.voice.permission' : voice.phase === 'recording' ? 's16.voice.recording' : voice.phase === 'transcribing' ? 's16.voice.transcribing' : voice.phase === 'loading' ? 's16.voice.loading' : 's16.voice.playing')}} +
+ {!voice.status ? t('s16.voice.checking') : !voice.status.available ? t('s16.voice.unavailable') : t('s16.voice.local')} + {usage && {t('s16.voice.usage', { used: (usage.chargedMs / 1000).toFixed(1), reserved: (usage.reservedMs / 1000).toFixed(1), limit: (usage.limitMs / 1000).toFixed(0) })}{usage.conservativeMs > 0 && <> {t('s16.voice.conservative', { seconds: (usage.conservativeMs / 1000).toFixed(1) })}}} + {voice.phase === 'draft' &&