Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions devlog/_plan/260912_audio_apis_stack/000_plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Expose file transcription, streaming dictation and GPT-Live to external clients

Owner steering during wp1 C: no local Bun suites, product tests, typecheck, build or dependency installation. Push with --no-verify and use exact-head remote CI for remaining executable verification. This supersedes every local command example in the layer plans. Already completed checks are historical evidence only; interrupted/crashed checks are not passing evidence. All active task-owned local suites were stopped. Functional layer closure uses the completed source review and pre-restriction focused evidence; PR readiness and final completion retain the remote CI gate under wp3 publication.

Owner scope steering during wp2: finish the audio stack and record pre-existing unrelated CI failures separately. Do not extend this task into further journal-restore or CLI stale-process repairs. Audio-owned regressions and source-review blockers still require closure. The already-published prerequisite corrections remain in the bottom branch; their remote outcomes are reported honestly. Aggregate CI failures are not represented as passing checks, and PRs remain drafts where baseline failures prevent full readiness.

- Archetype: satisfy-spec, C4 API/auth and C3 dashboard integration.
- Trigger: owner requested both audio capabilities, inherited subagent verification and a published dependent PR stack.
- Goal: three independently reviewable ordinary PRs with protocol tests, documentation and usable client controls.
Expand Down
24 changes: 23 additions & 1 deletion devlog/_plan/260912_audio_apis_stack/020_streaming_voice.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,27 @@

Depends on wp1 audio upstream resolution. Preserve existing Codex transport exports and native integrations.

P revalidation at 156e28f4ba: prior D completed the functional transcription layer and retained final-head CI as a publication gate. Production interface now resolves AudioUpstream with admission/model/lease/exactAccountId/signal and owns explicit-key admission before loopback fallback. Remote-only owner policy supersedes local test commands below; implement tests but run them only in CI.

Dependency repair: remote run34684544174 identified pre-existing Cline count/expectation/provenance gaps and a preimage-read injection fixture whose interception was not observed. The bottom branch carries narrow corrections reviewed independently, with all original preimage assertions retained and an added interception counter. This upper branch must merge the updated parent before publication. No local product checks run; the fixture hypothesis is validated only by new remote CI.

## File changes

| Operation | Path | Contract change |
| --- | --- | --- |
| NEW | src/server/audio-dictation.ts | exact streaming route, canonical upstream URL/protocol construction and bounded session policy |
| NEW | src/server/audio-client.ts | audio-only browser credential carrier, explicit key detection and tagged owner identity |
| NEW | src/server/live-call-bindings.ts | per-server bounded expiring call ownership, keyed by opaque call id and admission owner |
| MODIFY | src/server/live.ts | bind successful call-create to its resolved upstream; join via binding; proxy Bearer credential support; default V3 model/negotiation only on standalone /live when absent |
| NEW | src/server/audio-live.ts | external keyed call-create/join, account binding, safe Location and default standalone V3 negotiation; reuse Live protocol primitives |
| MODIFY | src/server/live.ts | expose the existing multipart conversion primitive for the external audio owner; preserve native handler behavior |
| MODIFY | src/server/index.ts | route dictation upgrades through existing bounded WebSocket bridge; thread lifecycle metadata and close cleanup |
| MODIFY | src/server/ws-bridge.ts | add only necessary session protocol/expiry fields to WsData |
| MODIFY | src/server/auth-cors.ts | advertise only HTTP call-create rows; WebSocket admission is separate audio metadata |
| MODIFY | tests/server/api-key-attribution.test.ts | valid SDP multipart fixtures for HTTP call-create matrix rows; WebSocket auth is tested by real upgrades in audio/server-live tests |
| NEW | tests/server/audio-dictation.test.ts | mock WebSocket upstream with real JSON audio events and close/cancel checks |
| MODIFY | tests/server/server-live.test.ts | external Bearer key and call-owner/session lifecycle regressions |
| NEW | tests/server/audio-client.test.ts | browser carrier, key precedence and stable key-rotation ownership |
| NEW | tests/server/live-call-bindings.test.ts | opaque call aliases, expiry, capacity and Location parsing |
| MODIFY | scripts/test-layout/layout.json | register dictation test in server domain |
| MODIFY | tests/fixtures/test-layout-expected.json | matching expected test path |
| MODIFY | structure/data-planes/inbound-compat.md | document streams, ownership and source/test contract |
Expand All @@ -28,6 +36,14 @@ Client uses observed session.start/config, audio append and session.close shapes

Observed dictation audio is JSON {type: "audio.append", audio: "BASE64_PCM16"}; mono PCM16 uses the actual sample_rate_hz supplied by the client. transcript.segment/final revisions replace prior text for the same utterance_id. Closing acknowledgment is session.updated with session.status=closed. The requested limits are client policy, not proven upstream maxima. For browser clients the downstream protocol pair is opencodex-audio plus opencodex-key.<proxy-key>; only opencodex-audio can be selected back. Explicit HTTP admission headers retain precedence. Never accept this carrier on ordinary Responses routes.

The protocol carrier suffix is canonical base64url of the UTF-8 proxy key, not a raw key; encoding is transport syntax, not encryption. Bound header/token sizes, reject duplicates and mismatched marker/key pairs, preserve explicit HTTP-header precedence and use only opencodex-audio as the selected downstream protocol. WsData receives upstream protocols through Bun.WebSocketOptions.protocols (verified in installed bun-types), plus optional validation/lifetime callbacks. Clear retained handshake credentials after connecting. This new field is constructed at upgrade, consumed by attachLiveSidebandUpstream and cleared on close; it never enters JSON persistence.

The per-server registry stores only tagged owner identity, provider ID, account ID/physical account identity, endpoint policy and expiry. Configured-key owner identity uses key ID (rotation preserves it); environment identity uses a digest of the verified admission token. Proxy-key-created calls reject unknown, expired, mismatched owners before selecting credentials. Resolve the recorded account freshly and verify physical identity before joining; caller-owned native context must be resupplied. Do not keep credential snapshots as a substitute for ownership. Bound capacity at 1024 and TTL at 30 minutes; pruning is demand-driven and shutdown clears the map.

Module-responsibility refinement: external keyed live HTTP/WS orchestration belongs in audio-live.ts so the native compatibility handler retains its existing behavior. index.ts selects this owner only after audio-client resolves a verified explicit key. Shared multipart conversion, URL builders and protocol header names stay owned by live.ts. The external create path has a lease-bound operation controller, upload/overall deadlines, one final outcome, and content-free failure responses. A bound join restricts provider selection to the recorded provider as well as recorded account; a newly enabled provider cannot displace the call owner. Opaque sideband close reasons are replaced with a generic reason for external audio clients.

A audit fold-back: transfer AudioUpstream.release and recordOutcome into a once-only WsData.liveFinish callback only after server.upgrade succeeds. Before-transfer refusal/exception releases the acquired context and lease directly. The bridge calls liveFinish exactly once from finalizeLiveSideband when the upstream is observed CLOSED (or no upstream was created), then releases the turn lease in finally. Socket construction/open/error/timeout/close paths carry a single terminal outcome; client cancellation before connection is neutral. Clear handshake/session timers and retained headers/protocols on all finalization paths. Do not release ownership merely because downstream closed while upstream remains CONNECTING/CLOSING.

Live call-create keeps SDP/multipart conversion and Location response. A successful call stores a bounded per-server binding to the resolved account/provider and caller admission identity for follow-up joins. Never retain raw client API keys in persisted state; no persistence is needed. Follow-up requests authenticate again, reject mismatched/expired owners, and cannot change the selected account. Existing native clients using the same local/session identity retain their workflow. Validate invalid Location before reporting usable creation.

Bindings survive sideband disconnect for bounded reconnect; server shutdown clears them. Resolve the recorded exact account before join, preserving physical account identity across credential refresh. Tagged ownership distinguishes configured key ID, environment admission and legacy loopback native session. Unknown calls are rejected for proxy-key clients; any native externally-created-call compatibility must remain limited to explicit caller-auth and documented separately. Reuse the existing socket bridge within the composition root for this layer; extracting all legacy socket machinery is optional and requires its own regression evidence.
Expand All @@ -36,6 +52,12 @@ Live resolves a presented proxy key before the global loopback shortcut, so a ke

For external proxy-key call-create, return a proxy-relative Location /v1/live/<validated-call-id> (or the matching realtime/calls form). Never return an absolute upstream Location to a proxy-key client. The upstream WebSocket destination is independently selected from trusted config, not from Location. Native legacy response compatibility remains scoped to its existing explicit caller-auth contract. A regression follows the returned relative Location with the creator key and checks the recorded account after pool rotation.

The externally returned call ID is a proxy-generated rtc_ocx_<random> alias mapped to the validated upstream call ID, retaining compatibility with rtc_-accepting clients. The reserved rtc_ocx_ namespace never falls through to legacy joins, including after expiry/removal. This prevents an expired external alias from being reclassified as an unowned native call. The binding also preserves the originating Frameless/realtime join style; aliases do not let the caller change the upstream protocol family.

Keyed upstreams have no verified physical account ID, so their bindings additionally retain a digest of the upstream credential. Fresh join resolution compares that digest and refuses a changed credential instead of guessing that a replacement key owns the original call. This digest is created at call registration, copied only in the in-memory registry, consumed at join and removed at expiry/shutdown; it is not serialized or exposed.

Source-review refinement: normal WebSocket completion is neutral for account health because transport open can be followed by a protocol rejection. Only explicit transport failures/timeouts reach the recorder. Native platform-bearer compatibility uses equality with the configured canonical OpenAI API credential, never an sk- prefix guess; unrecognized/revoked custom bearers are rejected. The existing global listener policy still governs whether a native request is admitted without a proxy key.

Standalone WS /v1/live accepts an explicit model or defaults to gpt-live-1-codex, with gpt-live-1 as documented alias if implemented. Missing V3 negotiation is added only to this Frameless path. /v1/realtime preserves its current adapter semantics. A raw live client receives delegation events; proxy does not execute tools or fabricate delegation results.

WsData fields are created at server.upgrade, serialized only by Bun in process, read at open/message/close, and disposed at relay closure; no disk reviver. Call binding types are in-memory only. Public query/model and protocol inputs are validated at ingress; no secret-bearing URL query authentication is added.
Expand Down
2 changes: 2 additions & 0 deletions docs-site/src/content/docs/fr/reference/proxy-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,8 @@ Voir [le guide Desktop](/fr/guides/claude-code/). Relecture thinking et cache re

## `POST /v1/live` et bande latérale en temps réel

La liaison de compte ci-dessous concerne les clients Codex natifs. Pour la dictée et GPT-Live avec une clé API externe, consultez la [spécification audio en anglais](/reference/proxy-formats/#streaming-dictation).

`POST /v1/live` accepte la surface de création d'appel ChatGPT/Codex App sans cadre.
`POST /v1/realtime/calls` accepte la surface de création d'appel OpenAI Realtime. opencodex sélectionne un
route OpenAI-family éligible, normalise la demande de création d'appel pour l'authentification en amont
Expand Down
2 changes: 2 additions & 0 deletions docs-site/src/content/docs/ja/reference/proxy-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,8 @@ API ではありません。Desktop のキー移行・復旧・切断は既存

## `POST /v1/live` とRealtime サイドバンド

以下のアカウント連携は既存の Codex クライアント向けです。外部 API キーで利用する音声入力と GPT-Live は[英語版の音声 API 仕様](/reference/proxy-formats/#streaming-dictation)を参照してください。

`POST /v1/live` は、ChatGPT/Codex アプリのフレームレス通話作成サーフェスを受け入れます。 `POST /v1/realtime/calls` は、OpenAI Realtime 呼び出し作成サーフェスを受け入れます。 opencodex は、適格な OpenAI ファミリ ルートを選択し、アップストリーム認証モードのコール作成リクエストを正規化し、制限付き応答を中継します。

コールの作成後、クライアントはサポートされている受信フォームを使用してサイドバンド WebSocket に参加できます。
Expand Down
2 changes: 2 additions & 0 deletions docs-site/src/content/docs/ko/reference/proxy-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,8 @@ HTTP 400을 반환합니다. 두 경우 모두 날짜 제거나 다른 경로로

## `POST /v1/live`와 Realtime sideband

아래 계정 연결 설명은 기존 Codex 클라이언트 기준입니다. 외부 API 키로 쓰는 받아쓰기와 GPT-Live는 [영문 음성 API 명세](/reference/proxy-formats/#streaming-dictation)를 따릅니다.

`POST /v1/live`는 ChatGPT/Codex App Frameless call-creation 표면을 받습니다.
`POST /v1/realtime/calls`는 OpenAI Realtime call-creation 표면을 받습니다. opencodex는 적절한 OpenAI 계열
경로를 선택하고, 업스트림 인증 모드에 맞게 call-creation 요청을 정규화한 뒤, 제한된 응답을 릴레이합니다.
Expand Down
Loading
Loading