Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ S15/S16 target terms below refine product intent. Definitions explicitly marked

**Shared client (S16, API16-01):** The browser/native-safe transport and ephemeral projection state over the existing protected Jit service. It submits exact owner commands, reads currently authorized snapshots/control events, recovers the original send identity and reviews exact actions. It is not a second authority, engine or store. The [client contract](docs/contracts/s16-client-v1.md) supports text/tool work and exact decisions. Explicit file composition additionally supports the first [UTF-8 file-to-artifact workflow](docs/contracts/s16-files-v1.md) and React `/s16` client; an explicit strict parser configuration additionally supports [isolated PDF/PNG/JPEG inputs](docs/contracts/s16-binary-files-v1.md) through actual guarded Schway image sends. Explicit [sourced file memory](docs/contracts/s16-file-memory-v1.md) additionally carries exact original/representation pins through later work and inference history, with withdrawal/correction/forget and current recipient/private-scope checks; live steering remains unavailable; Linux isolation and live-account vision evidence remain separate pending acceptance.

**Voice draft (S16, VOICE16-01):** A finite owner-initiated microphone clip and editable transcription under explicit protected-host local speech configuration ([ADR 0096](docs/adr/0096-s16-bounded-local-dictation-and-playback.md)). Capture reserves local audio-second quota, not Bot work. The transcript enters CHAT only through ordinary Send; spoken approval words cannot decide an action. Cancel discards the draft independently of work. Only a currently published reply with exact conversation/request/revision/text pins can be synthesized. Interrupt stops client audio, without implying a Bot or effect stopped. Raw audio/drafts are ephemeral; bounded metadata-only quota survives restart with conservative unknown charges and no speech replay. The [voice contract](docs/contracts/s16-voice-v1.md) does not deliver #423 continuous calls or #431 physical-device acceptance.

**File lineage pin (S16, MEM16-01A):** Immutable metadata naming an exact original source/resource generation and original/validated-representation hashes. MEM/TEAM context, claim and artifact ancestry retains these pins; owning FileStore status and selected bytes are rechecked before release. A pin is data, never a principal or permission. Explicit opt-in freezes new file-memory ingress; old accepted file commands retain their prior semantics.

**Work item:** A proposed or owner-requested obligation with a status, explicit completion rule, origin and evidence. A work item may involve multiple Bots and conversations without becoming a Project.
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ Project access levels, scripted replies, `jit ask`/`pending`, signed webhooks an
| --- | --- |
| [Service install and upgrade](docs/runbooks/ops-service.md) | Build, plan, install and upgrade the systemd service |
| [S15 install](docs/runbooks/s15-install.md) | Opt-in S15 installation (`install.sh --s15`), upgrade, rollback, uninstall and owner steps |
| [Local voice](docs/runbooks/s16-local-voice.md) | Explicit local dictation/playback, pinned speech artifacts, finite usage and opt-in proof |
| [Execution box](docs/runbooks/execution-box.md) | Rootless Podman setup and `box-doctor` |
| [Pi live round](docs/runbooks/pi-live-round.md) | Host, subscription and reboot evidence on a Pi |
| [Load drill](docs/runbooks/ops-load-drill.md) | Capacity and release-evidence drills |
Expand Down
63 changes: 63 additions & 0 deletions docs/adr/0096-s16-bounded-local-dictation-and-playback.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
status: implemented optional local dictation/playback; installation and full voice acceptance remain separate
date: 2026-11-18
scope: VOICE16-01 (#422); S16-AC15; FR-06/11/13/14/19/24/25/37/38; NFR-05/06/07/09/11
---

# Bounded local dictation and reply playback

## Required outcome and current baseline

S16-AC15 and the authored [voice experience](../design/S16-rakazo-experience.md#voice-and-clients) require intentional microphone capture, visible recording/transcription, an editable transcript, cancellable playback and finite usage. Text remains usable. Audio/transcribed approval words cannot invoke an exact protected decision. Continuous calls belong to #423; native account/device evidence belongs to #431. The primary agent owns implementation/design under the current AGENTS agreement.

At main `be21d747d25f4a6a234f057d8b35362501b03a7f`, the shared client (#395) and connections (#417) prerequisites are merged/closed. The React composer, guarded owner projections, command identities and exact decision cards are available. There is no audio implementation; `tests/chat-voice*` tests prose style. #422 was unassigned with no competing branch/PR; it is now owned on an issue-specific worktree. Existing worktrees are untouched.

## Alternatives and source observations

| Choice | Benefits and limits | Selection |
| --- | --- | --- |
| Browser/system speech recognition and synthesis | Little server infrastructure. Recognition may use a browser/vendor service or lack a language/platform; synthesis voices can also be remote. A system API is not evidence of local custody | Do not silently use it as local speech. A future explicit device-local adapter must prove its locality and availability |
| Protected provider STT/TTS | Broad language/voice coverage; requires exact account/recipient, billing scope, current private-source checks and uncertain-usage handling | Keep as a future adapter option, not an automatic fallback or a prerequisite for configured local voice |
| Local CLI STT/TTS on the protected Jit host | Actual speech without a new external recipient or paid account. Operator supplies verified binaries/model; resource limits and lifecycle are Jit's responsibility. Recognition/voice quality varies | Select `whisper.cpp` transcription and `espeak-ng` synthesis on Linux, with an explicit `say` synthesis adapter for macOS development/evidence |

The selected local route satisfies #422's configured-local-or-provider choice. It does not claim every provider or device is supported. The host uses fixed argument arrays, private bounded files and finite processes, never a model-provided command or a working-VM execution backend. Schway continues to own model/tool execution; this owner-client audio adapter introduces no provider loop or private Schway import.

Rakazo inspection: repository `elie222/rakazo`, immutable commit `abf3c08145a05cba92a912d6c24fdfb5f569d0e0`, tree `f8ca5adc96366b722bb60186c21cb29cec0f5390` (complete tree, not truncated). Read without executing upstream scripts:

| Source | SHA-256 | Observation |
| --- | --- | --- |
| `packages/adapters/src/openai-compatible-voice.ts` | `33c9fd666df55f36305b26e1a0f9b6deaf0e73e36e07e0382c64f51d5f154c76` | Explicit compatible server/model credentials, bounded transcription/synthesis calls; not Jit's recipient/budget authority |
| `packages/adapters/src/voice-http.ts` | `ceda3d36f626a5893e808f8e94ceb2edd7ff853a234c718165697843359fb80e` | Finite request deadlines and response byte limits are useful concepts |
| `apps/desktop/native/speech/main.swift` | `099d8862dae69ebbd077b352c30afcbf21e949c7008cba1c08c56f0225295b30` | Requests on-device recognition when supported, otherwise permits system service; that fallback cannot establish Jit's local-only route |

These are comparison observations, not upstream vulnerability claims. No upstream source/assets/dependencies are copied. Existing Apache-2.0 inventory/notices remain unchanged; future copied files follow ADR 0074/0080. CLI binaries, voice data and model weights are separately operator-supplied artifacts; each installation must retain their applicable licenses/provenance rather than infer them from Jit's MIT license.

## Selected ownership and custody

The implemented [s16-voice/v1](../contracts/s16-voice-v1.md) owns finite client capture sessions and local speech usage. It adds no Bot tool, message admission or approval authority. The configured voice route is installation-owned, opt-in and contains no credentials. Its artifact paths/hashes are protected configuration; clients get labels, capabilities and limits only. No automatic binary/model download or installation activation occurs.

The owner starts dictation with a deliberate Mic press. The authenticated service reserves finite local quota before capture. The browser then requests microphone permission, captures bounded PCM and stops every track/audio node on Stop, Cancel, permission loss, page/target/session change or deadline. Cancelled/late permission promises immediately stop any subsequently supplied stream. No browser speech-recognition service is contacted.

Stop submits a bounded canonical WAV to the same authenticated Jit service for one fixed local transcription. A transcript is untrusted draft data. It is not a conversation message until the owner uses ordinary Send; editing and cancellation create no Bot work. Spoken/quoted "approve" cannot call `decide`, `decideAction` or change purpose. Existing file-bearing/steering semantics remain explicit rather than being inferred from speech.

Playback selects the exact currently guarded reply by conversation/request/revision/text digest; it does not accept arbitrary unguarded text from the page. Recheck owner/session and source publication before local synthesis and before publishing audio. Interrupt stops playback immediately and fences pending playback responses; underlying Bot work keeps its existing state. Local synthesis is not evidence that an effect succeeded.

## Bounds, uncertainty and compatibility

Use finite per-capture duration, WAV bytes, transcript/reply bytes, local process time, synthesized audio duration/bytes and persistent per-day local speech quota. Configuration can narrow published hard limits. Unknown/failed process usage retains its conservative reservation; do not reset quota on restart or confuse audio seconds with measured CPU/GPU time, provider tokens or a zero monetary bill. Local route loss/budget exhaustion ends voice with a safe code while ordinary text remains usable. No fallback creates a new account/recipient or raises limits.

Review refinement: recording and submission have separate deadlines. Slow microphone permission can consume the original grace, so the capture receipt exposes `recordingEndsAt` and a later finite `expiresAt` with 15 seconds for cleanup/upload. The browser never records into that submission window; the service accepts a completed bounded clip until submission expiry. Actual subprocess barrier/settlement JSON is published by atomic rename after the complete file is written, so file existence cannot expose a partial receipt.

Audio/transcript scratch is private and short-lived, never in model prompts, working VM, exports, raw errors or logs. Submitted text uses existing conversation retention. Metadata-only quota receipts remain protected; abandoned pending operations are terminal after restart and cannot silently replay. Shutdown kills and actually joins local children before removing scratch/releasing capacity. Browser stop and server settlement are separate facts.

This is a new optional voice journal/port, not a migration of CHAT, TEAM, MEM, FILE or policy meaning. S14 stays default; S15 opt-in and omitted-runtime compatibility stay. No existing owner host/account/policy is changed by implementation. Packaging verifies the optional feature without bundling a model/binary or asserting device acceptance. #422 closes only after actual composed client/service and native speech evidence plus required checks/review; this document alone does not complete it.

The [operator runbook](../runbooks/s16-local-voice.md) describes explicit configuration, artifact licenses/pins, usage and recovery. Runtime source reaches the shipped package through the existing `src` inventory, including the independent supervisor and fixed worklet. No new package dependency or imported upstream asset is required. Health metadata caching is descriptive only: each actual speech job rehashes executable/model pins. A valid bounded macOS `say` WAV may include `FLLR` padding; synthesis validation accepts it without weakening the canonical capture format.

[Local speech evidence](../evidence/s16-local-voice-proof.md) records actual macOS/Linux ARM64 speech, the real English/Thai bundle, process-crash/unknown-usage fixtures, exact artifact hashes, failed experiments and remaining acceptance. Linux eSpeak NG required the separately pinned `base.en` model to pass the unchanged known-speech assertions; the smaller `tiny.en` failure is retained as a quality limit. This is not full continuous-voice/device acceptance.

## Verification obligations

Use deterministic PCM/command fixtures, fake clocks and barriers for quota, cancellation, lost responses, stale generations and current owner/publication checks. Browser tests exercise the real bundle in English/Thai and mobile/keyboard layouts, with fake media devices rather than a real microphone. Native opt-in evidence must transcribe actual known speech and synthesize an actual guarded reply through configured local executables, recording exact binary/model/build hashes. Label recognition accuracy, device permissions and all untested platforms honestly. #423 retains continuous-call/barge-in/reconnect behavior.

Protocol references inspected 2026-11-18: [Media Capture and Streams](https://www.w3.org/TR/mediacapture-streams/), [whisper.cpp CLI implementation](https://github.com/ggml-org/whisper.cpp/blob/master/examples/cli/cli.cpp), [espeak-ng command reference](https://github.com/espeak-ng/espeak-ng/blob/master/src/espeak-ng.1.ronn). Implementations must validate their exact supplied build; these moving references are not installation pins.
Loading
Loading