fix(chat): preserve JSON completion semantics in streaming delivery

devlog/_plan/260907_track2_protocol/000_plan.md

@@ -0,0 +1,28 @@
+# Track 2 protocol delivery
+
+- Archetype: satisfy-spec repair with an evidence-backed defer outcome.
+- Trigger: maintainer assigns track 2 and authorizes ordinary PR chains, no-verify pushes, final remote CI first, and admin integration.
+- Goal: preserve Chat JSON/SSE semantics (#3770/#3779), refusal (#3767), supported custom efforts (#3775), hosted-search execution (#3761), and opt-in Claude compatibility (#3730), or document a concrete unresolved blocker.
+- Non-goals: native GitHub stacks, local tests/typechecks/builds/install, other tracks, service/config changes outside repository, releases/deployments. Do not weaken CI definitions or treat skipped tests as passing.
+- Baseline: dev 7d8523eed75a67f7a4a15b533744fcd0e6059aa8, including #3771.
+- Verifier: existing workflow_dispatch ci.yml lane=all at the final integration head; lower diagnostic CI only if final fails. Commands are NOT RUN locally by explicit instruction. Read workflow definitions to establish target coverage. Independent source review precedes remote execution.
+- Stop: feasible reviewed changes land through dev PRs with verification evidence; other items receive explicit evidence-backed dispositions. No completion claims for deferred issues.
+- Artifact: this numbered unit; private security analysis and raw tool results only in /tmp/cf54-*.
+- Expected outcomes: landed, already implemented, deferred with concrete blocker, or blocked by external CI/service state.
+- Escalation: parent reclaims failed worker slices; never widen auth/routing trust or retry provider work to make a test pass. No user budget was set.
+
+## Roadmap
+
+1. Docs-only roadmap and independent audit.
+2. 010: JSON Responses to streaming Chat semantics; carry #3779 with attribution.
+3. 020: refusal across live SSE, final snapshots, JSON, collection, and JSON-to-SSE.
+4. 030: custom effort provenance/capability repair after independent Codex source check.
+5. 040: hosted-search path feasibility, then scoped execution/continuation repair or defer.
+6. 050: opt-in Claude compatibility gate after independent official-contract/security audit or defer.
+7. 060: final source audit, remote CI, ordinary PR-chain integration and exact dev ancestry proof.
+
+The semantic stack is JSON fallback -> refusal. Catalog and Claude slices have disjoint implementation owners and join the integration tip. Source refs are ordinary branches, not registered native stacks. Do not run a separate lower-level CI before the final integration failure.
+
+## Process availability
+
+Installed cxc skills resolved to 0.2.20 because the named 0.2.19 directory is absent. No SessionStart binding was injected into this task; SESSION-IDENTITY-01 forbids borrowing a prior/transcript id. Therefore no FSM activation is claimed. Durable P/A/B/C/D artifacts and the native active goal still track authorized work; tests remain pending until remote evidence exists.

devlog/_plan/260907_track2_protocol/010_chat_json_sse.md

@@ -0,0 +1,22 @@
+# JSON Responses to streaming Chat
+
+Depends on roadmap only. Class C3; PR #3779 is the public implementation source.
+
+## File delta
+
+- MODIFY src/server/chat-completions.ts:455: keep responsesJsonToChatCompletion as semantic authority. Replace text-only extraction and constant stop with converted choice.message content/reasoning_content/refusal; project tool_calls with stable array-order indices; preserve converted finish_reason. One role event, at most one combined delta, one terminal and one DONE. No extra upstream inference.
+- NEW tests/responses/chat-json-sse-fallback.test.ts from the source PR: actual loopback Responses upstream -> handleChatCompletions. Include one/two tools, reasoning+text, incomplete length, empty completion, cancellation and translator-budget release.
+- MODIFY scripts/test-layout/layout.json and tests/fixtures/test-layout-expected.json: register new test under responses.
+- MODIFY docs-site/src/content/docs/reference/proxy-formats.md and structure/04_transports-and-sidecars.md: buffered fallback delivery, semantic parity and no additional request.
+
+## Activation and oracle
+
+Streaming Chat request + JSON Responses upstream is the trigger. Native Chat and real SSE bypass this path. Hardcoded official Chat fixtures require indexed function calls, nullable finish for intermediate chunks and original terminal finish. First-choice scope follows the existing Responses single-result contract. Official openai-node ChatCompletionChunk/ChatCompletionMessage are the independent shape oracle; source PR tests are evidence candidates, not a passing result.
+
+## Check and delivery
+
+No local execution. Final remote CI must execute tests/responses/chat-json-sse-fallback.test.ts and existing chat-completions-endpoint coverage, typecheck and test-layout guards. Preserve upstream author credit in carried commit and final PR body. Lower refs are published with --no-verify; no native stack registration.
+
+## Build checkpoint
+
+Carried #3779 and applied independent-audit corrections: shared native serializer, indexed tools, typed unknown incompletes, correct length/content_filter precedence, and explicit converted/serialized byte ownership. New tests preserve the source PR cases and add official-contract boundary/accounting cases. Local suites/typecheck/build NOT RUN by instruction; git diff --check is a whitespace check only. Remote verification remains pending.

devlog/_plan/260907_track2_protocol/011_chat_audit_amendment.md

@@ -0,0 +1,15 @@
+# Chat plan audit resolution
+
+Independent reviewer Fermat returned GO-WITH-FIXES, five blockers. No local command was executed.
+
+1. Accepted: source oracle differs from current converter. Incomplete max_output_tokens -> length and content_filter -> content_filter take precedence over tools. Incomplete missing/max_messages/steered/other reason becomes typed upstream truncation rather than fabricated token exhaustion, matching existing live-SSE handling of unknown incompletes. Both JSON and SSE public handlers map typed error, without success DONE.
+2. Accepted with scope: use existing budget owner; add optional budget to pure converter so runtime caller charges retained copied content/reasoning/tools. Reuse existing native jsonCompletionSse owner for both JSON-to-SSE routes after it gains optional budget-aware serialization and proper final tool indexes. Charge serialized strings and output buffer while simultaneously live; release temporaries on transfer, retain response bytes until consumption/cancel finalization. No general translator-budget refactor. Positive charge and small configured-budget overflow tests required; no local runs.
+3. Accepted: refusal parts indexed by original output_index/content_index; item.id/item_id are optional correlation constraints and a present mismatch fails. Preserve original array positions. Buffer refusal parts until terminal and emit in output/content order, avoiding interleaved-part reordering. Deltas append; equal snapshots deduplicate; extending snapshots fill suffix; shorter-prefix/empty snapshot preserves known data (sparse compatible provider); absent field is no new evidence; explicit non-string/contradictory non-prefix snapshot fails. Budget text plus per-entry metadata/key bytes, release on terminal/cancel/fail. Zero-length parts cannot bypass map accounting. Final JSON and collector use nullable refusal field.
+4. Accepted: native JSON-to-SSE uses same shared helper and preserves refusal. Matrix covers native/translated upstream JSON/SSE with client JSON/SSE. Native streaming passthrough remains opaque.
+5. Accepted: collectChatCompletion catch cancels reader before releasing lock, invoking upstream translator cancel; tests prove cancellation under processing overflow and no successful partial JSON. Existing outer budget finalizer remains final response owner.
+
+These replace conflicting portions of 010/020. Re-audit before source edits. Official SDK source field definitions retained in /tmp/cf54-openai-responses-types.ts and /tmp/cf54-openai-chat-types.ts. Aside page-open reached its host deadline without content; no browser-source proof claimed.
+
+## Re-audit resolution
+
+Fermat re-audit accepted the five resolutions and found one remaining terminal-order blocker. Accepted: stage all final role/tool/refusal/finish/DONE frames as one bounded terminal batch; serialize and reserve every frame before enqueueing any success frame. On reservation failure, release the staged reservations/refusal state and emit only the bounded typed overflow error (no success finish or DONE). Commit terminated success only after batch admission. Merely moving the terminated assignment is insufficient. Add a small-budget fixture that fails at final batch admission and asserts absence of success finish/DONE plus typed error and cancellation.

devlog/_plan/260907_track2_protocol/020_refusal.md

@@ -0,0 +1,17 @@
+# Preserve refusal across Chat projections
+
+Depends on 010 for JSON-to-SSE field delivery. Class C3 public wire contract.
+
+## File delta
+
+- MODIFY src/chat/outbound.ts: responsesJsonToChatCompletion accumulates content parts with type refusal and their refusal string into message.refusal, alongside existing content/reasoning/tool fields.
+- MODIFY same file live translator: map response.refusal.delta to delta.refusal. Track each output/content part separately with the existing translator budget. Final response.refusal.done, content_part.done, output_item.done and completed/incomplete snapshots may add only an unseen matching suffix. Repeated final representations must not duplicate text. Conflicting snapshots cannot be represented as append-only deltas and must terminate as a typed translation failure, not false success. Map storage and release follow existing turn-budget lifecycle.
+- MODIFY collectChatCompletion: collect delta.refusal with retained_collectors budget and serialize message.refusal. Never coerce refusal into ordinary assistant answer text.
+- NEW tests/responses/chat-refusal.test.ts (register both test-layout inventories): cover direct JSON, split live deltas plus all snapshot representations, done-only, terminal-only, multiple parts, mixed text/refusal, stream collection, contradictory snapshot failure, cancellation/overflow, and JSON-to-SSE handler path inherited from 010.
+- MODIFY proxy-formats.md and structure/04_transports-and-sidecars.md: document refusal field and parity across delivery shapes without claiming a new policy decision.
+
+## Independent oracle and acceptance
+
+OpenAI Responses docs define refusal.delta.delta and refusal.done.refusal, indexed by output_index/content_index; official Chat SDK defines delta.refusal and message.refusal. Local official Codex corpus is read only for consumer behavior; it is not automatically the sole Chat API schema authority.
+
+Trigger known refusal events/parts, expect exactly one concatenated refusal, unchanged normal content/tool semantics, one terminal+DONE on valid completion, typed failure without success DONE on invalid final snapshot or overflow. Tests use inert fixture messages rather than provoking live model refusals. No local suites run; remote final CI owns executable proof.

devlog/_plan/260907_track2_protocol/030_custom_efforts.md

@@ -0,0 +1,18 @@
+# Proven custom native capability projection
+
+Depends on roadmap; independent from Chat semantics. Class C3. Issue #3775 remains partial because an arbitrary gateway model name does not prove native capability, and current official source is not a binary proof for Desktop 0.153.4.
+
+## File delta
+
+- MODIFY src/codex/catalog/provider-fetch.ts: in current custom-row producer, retain existing canonical openai forward destination and capability-backed model ID proof. AFTER custom/inherited metadata merge, intersect explicit reasoningEfforts with nativeReasoningEfforts for that proved model. If explicit [] preserve [] and remove default; if nonempty declared list has no supported entry, use proved native default as singleton. Otherwise choose declared default only if present, then proved default if present, then first surviving effort. Do not clamp arbitrary provider/model names, destination overrides, or ordinary routed custom models.
+- MODIFY src/codex/catalog/sync.ts: in retained sync merge, current invocation's live custom rows must not have max re-added. Use current config/producer provenance rather than a disk marker. Keep ordinary provider/combo/Reserve rules.
+- MODIFY existing catalog-custom-models, sync-hardening, convergence and Claude model-discovery tests after reading actual filenames: canonical Astra with none/minimal + valid ladder; explicit []; all-invalid nonempty; valid default preserved; same-name noncanonical gateway unchanged; destination override unchanged; second sync no max resurrection; both gather entry points and /models client-version projection.
+- MODIFY relevant English catalog/reasoning reference plus structure/03_catalog-and-subagents.md to state the narrow capability proof and explicit empty-list behavior. Translations must not claim broader gateway/client repair.
+
+## Official evidence and deferrals
+
+Local official source corpus 121_openai-codex: protocol/src/openai_models.rs allows nonempty custom effort strings; models-manager/src/manager.rs qualified lookup is consumer lookup, not gateway provenance; multi_agents_common.rs validates chosen-row membership. Codex source supports ultra and translates it on wire. API model docs lacking ultra do not justify deleting Codex ultra.
+
+Field chain is existing custom config -> fetch custom row -> deriveEntry/retained merge -> catalog file/direct /models consumers. No new schema field or request-time effort override. Existing threads and version-specific stale runtime state are outside this repair.
+
+Source/code inspection is completed; proposed regression commands are NOT RUN locally. Final remote CI must exercise modified tests and typecheck. #3775 must remain open for gateway destination and exact Desktop/version evidence; this fix addresses only proven canonical rows.

devlog/_plan/260907_track2_protocol/040_hosted_search_disposition.md

@@ -0,0 +1,13 @@
+# Hosted search passthrough disposition
+
+Outcome: DEFER #3761; no production diff in this track.
+
+## Source findings
+
+src/web-search/loop.ts mutates normalized messages, while src/adapters/openai-responses.ts serializes passthrough _rawBody. Existing loop parsing is compaction-oriented and does not preserve the native tool/reasoning conversation needed for search-result continuation. src/server/sse-payload-rewrite.ts is synchronous rewriting rather than an asynchronous execution loop. Mixed ordinary tools/search and replay need a separate raw conversation contract; continuation storage alone does not supply it.
+
+Official Ollama local middleware supports hosted Responses search, including cloud model execution, but that does not establish the direct ollama.com/v1 endpoint contract. References opened during investigation: https://github.com/ollama/ollama/pull/17686 and https://docs.ollama.com/integrations/codex . Local official client source: corpus 121_openai-codex. No direct-cloud authenticated run was performed.
+
+## Resume criteria
+
+Define destination/backend execution policy; preserve raw Responses tools/reasoning when inserting search results; exercise mixed tools, cancellation, bounded iteration, SSE/JSON/WS, compact and replay remotely with a confirmed destination contract. A guard-only change is rejected because it cannot deliver these results. The issue remains open and will receive this disposition; no claimed fix, workflow or client change.

devlog/_plan/260907_track2_protocol/050_claude_compatibility.md

@@ -0,0 +1,23 @@
+# Claude compatibility carry
+
+Depends on roadmap and an independent source/security audit. Class C3 with C4 review of admission and persisted diagnostics. Public source is PR #3730 at 18d64748ade8001e05327726b2ae4b22e8393418.
+
+## Published scope and file map
+
+- NEW src/claude/compatibility.ts: opt-in analysis for translated Messages; no Lab imports, source envelope, credential access or adapter execution.
+- MODIFY src/server/claude-messages.ts: gate translated requests after real native passthrough returns and before inference. Preserve existing auth/origin and logging ownership.
+- MODIFY src/types/config.ts: compatibility mode property.
+- MODIFY src/server/request-log.ts and src/usage/log.ts: bounded optional metadata, persisted-row normalization and hydration.
+- NEW tests/claude-integration/claude-compatibility.test.ts and MODIFY existing endpoint/usage tests; register new filename in both test-layout inventories.
+- MODIFY server configuration reference and Claude guide only where current scope needs clarification.
+
+The source PR needs substantial classifier corrections before adoption. Detailed security-sensitive findings, official feature matrix, exact corrections and review evidence are held in task scratch space. The parent accepts a uniform conservative translated-path contract; no per-adapter exemption based on preliminary routing. Existing unset/native behavior remains unchanged. Shadow is observational and enforce is endpoint compatibility admission, not a global security boundary.
+
+## Field chain and acceptance
+
+Mode: typed config -> persisted JSON -> existing loader -> translated Messages gate. Present invalid mode must produce a fixed visible configuration failure rather than silently disable checking. No global config fallback change.
+Evidence: gate -> request context -> ring -> usage row -> normalized disk row -> hydration. Only closed protocol codes may be stored; no body/header/credential/signature material. Old rows remain valid.
+
+Test unset/shadow/enforce/invalid modes, real native bypass versus translated Anthropic, zero-inference rejected requests, normal tools versus hosted feature declarations, nested supported content positions, large headers, persistence and reload. Exact behavior follows the private reviewed feature matrix. No local test/typecheck/build/install; final remote CI and explicit independent security review required.
+
+Co-authored-by: SB Yoon <44089734+yansigit@users.noreply.github.com>

devlog/_plan/260907_track2_protocol/060_remote_delivery.md

@@ -0,0 +1,15 @@
+# Remote validation and delivery
+
+Depends on all accepted implementation slices. Class C3 integration; any admission changes require independent security review.
+
+## Delta
+
+- MODIFY only this unit's outcome/evidence record after source review.
+- Publish ordinary branches using git push --no-verify. Native stack registration and repo-wide workflow edits are excluded.
+- Final integration branch contains all accepted layers and current dev. Dispatch existing .github/workflows/ci.yml lane=all at its exact SHA; inspect every expected Linux/macOS/Windows shard and supporting gate. No local tests, typecheck, builds or install.
+- If final CI fails, inspect log/artifact and then use lower-head or bounded remote cases to isolate it. Never claim skipped/cancelled checks passed or silently weaken assertions. Existing automatic PR jobs may run; no fabricated status.
+- Open template-complete ordinary parent/child PRs; preserve contributor trailers. Record local NOT RUN, final integration proof, independently verified issue scope and explicit maintainer integration.
+- Merge accepted work through dev PRs with admin authority. Refresh actual base/head and maintainer objections before each write, retain parent refs while children target them, and prove each final merge is an ancestor of fetched dev. If dev changes in another track, review integration delta and refresh final proof where needed.
+- Close only fully resolved issues and superseded source PRs with attribution and replacement links; partial/deferred issues remain open with precise status.
+
+No source test is executed merely to verify that its command exists. CI workflow and package scripts are the source-inspection evidence of coverage. Report what each remote job actually did.