feat(kap-server): add the WS v2 realtime protocol stack (transcript-v3) - #3547
Open
liruifengv wants to merge 34 commits into
Open
feat(kap-server): add the WS v2 realtime protocol stack (transcript-v3)#3547liruifengv wants to merge 34 commits into
liruifengv wants to merge 34 commits into
Conversation
Single source of truth for the v2 wire contract: 25-type ServerMessage discriminated union, ClientMessage control frames, parseServerMessage, entityId/entityKey. Exposed as @moonshot-ai/kap-server/protocol/v2; code-app's app-core ws2/messages becomes a pure re-export.
Event2 -> ServerMessage projection core: per-agent timeline projector (turn/step/text/tool/prompt domains), session.state composer, and a script harness that deep-equals projected streams against the vendored 24-tab example fixtures (basic and tool tabs covered).
submitSteer enqueues through the same path as regular submissions, so a steer-bound prompt was indistinguishable from a queue-bound one on the event stream. The new optional steer flag lets projections hold the queued emission and place the user message directly into the active turn when prompt.steered arrives.
multi-tool, todo, queue-abort, tool-error and llm-retry streams now deep-equal the example fixtures. The projector gains: tools.update_store driven todo entities with projector-allocated td_* ids and the TodoWrite back-link, prompt.aborted driven system interruption frames (m_* ids), interrupted-step usage passthrough, error tool results without output, retry-step text reset with placeholder re-announce, and progress cleanup on tool completion. Turn usage now sums plain input_other/output only. ProjectionEvent declares its payload fields explicitly to satisfy noPropertyAccessFromIndexSignature.
…eams Six more fixture tabs now deep-equal: approval (both branches), question, background-task, compaction, undo and big-output. The projector gains a generic interaction path (pending frames with tool_call approval_id back-link, terminal frames preserving the original request), bus-driven approval events (permission.approval.requested/resolved), compaction system frames with summarized_through_turn derivation, undo frames from context.undone, shell.output driven task output_tail refreshes, and a turnId hint on prompt.submitted for engine-known turn clocks.
…treams Five more fixture tabs deep-equal (19 total). plan mode enter/exit edges from agent.status.updated with revision-tracked payloads, goal.updated system frames and goal-continuation turns without user frames, sidechat agents sharing the session turn clock, attachment_ids flowing from prompt submissions into user and turn frames, and cron.fired normalized through the prompt submission path. applyFacts now flushes in-flight text before composing session.state, and modes becomes a per-frame fact instead of sticky status state.
…live tabs injection (steer, busy-cron, busy-task-notify) and subagent (foreground, background, foreground-to-background, plus the on-demand subagent channel) now deep-equal the fixtures, covering every engine-synthesizable live tab. Busy cron injections take the system-steer path with pre-emptive completion ordering, subagent lifecycle drives task entities with child_agent_id and agent_refs on the parent tool call, and the subagent channel projects the child agent's own stream under an r-prefixed turn namespace.
recordRevision now derives a summary from the first markdown heading of the snapshotted plan so wire consumers can label revisions without fetching the blob.
buildColdHistory folds wire.jsonl into terminal v2 entities: turn groups of step groups in wire order, protocol ids reproduced cold, in-flight turns without a cover but with their completed steps, undo kept with a marker, clear/compaction as fold floors, and keyset pagination over before_turn/after_step. A 100k-line wire rebuild lands in ~240ms. The route is served through defineRoute and shares readColdWireRecords with the transcript service. Thirteen REST fixture sections now deep-equal the builder output alongside the twenty-one live streams.
A session binder feeds engine events to the v2 projector and routes the emitted messages to subscribed connections. WsConnectionV2 implements hello on connect, subscribe/ack with ErrorCode acks, per-type omit, a bounded outbound queue that closes slow consumers with backpressure_overflow, protocol-layer heartbeats, and on-demand subagent channels. Recovery is the same sequence as live: in-flight turn cover, current step, accumulated streaming entities, state entities, and a fresh session.state, synthesized from the projector's new recoveryEntities API. recovery A/B, omit, backpressure-reconnect and the subagent channel now deep-equal the fixtures end to end.
App-scope events now project to v2 global messages broadcast to every v2 connection: session index frames (meta.updated with SessionInfo and changed_fields, created/archived), workspace created/updated/deleted via toWireWorkspace, sanitized config with changed_fields, config.warning, and the thin model_catalog/plugin/capability notifications. Remaining system subtypes (hook, skill, notice, clear, swarm enter/exit) are projected, and the binder learns profile.bind as a model fact. The session-changes and global tabs deep-equal, and the full kap-server suite passes at 1374 tests.
Real-engine e2e (scripted provider, stub model) covers eight scenarios: basic text, yolo Bash, manual approval with the approval_id back-link, TodoWrite entities, abort with interruption, mid-turn disconnect recovery, REST history after completion, and v1 protocol coexistence. Fixes from the integration: engine step ordinals are 1-based so the projection normalizes to the 0-based wire ids (with a phase clamp before the first step starts), TodoList calls bridge to the TodoWrite wire vocabulary at both the live and cold boundaries, tools.update_store is now observable so live todo entities flow, and the wire manifest is regenerated for the plan.revision summary addition.
The session settles to idle before prompt.aborted projects the interruption marker; fixtures and the queue-abort script follow.
Durable interaction.request records and the interaction hub both carry engine-shaped payloads (toolName/action, question items with object options). Cold history and the live question path now pass them through a shared toWireInteractionRequest/Response mapper (approval requests recover their input from the linked tool call, question items get ids and string options), instead of leaking engine shapes onto the wire and tripping the client's contract guard.
PromptReservation.submit took no options, so a caller could not mark a submission as a steer at enqueue time and had to steer in a separate call afterwards. submit now accepts an optional steer flag and passes it through to enqueue.
POST /sessions/{id}/prompts accepts steer: true. The reservation is
submitted with the steer flag and, when the handle lands pending, is
steered immediately within the same request; a PROMPT_NOT_FOUND race
against an already-launched prompt is tolerated. Clients no longer
issue a separate steer call that rendered the steered user message
twice.
- skip empty thinking parts in the agent loop (live stream, interrupted drain, cold history) so no empty thinking shells are emitted - emit the steered prompt's user message when submission races with turn completion, assigning it to the new turn exactly once - emit goal system entries only on goal status transitions, and mirror that in cold history (goal.create produces the active entry, goal.update/clear produce none) - let before_turn history paging reach past the compaction floor so loading earlier messages keeps returning pages - refresh the v2 example fixtures and add a steer-race projection case
Approval requests in the v2 stream dropped the engine's ToolInputDisplay,
so an ExitPlanMode approval reached clients as an empty {} payload instead
of the plan review. The approval request schema gains a display field, the
live projector and the cold-history wire mapper pass it through, and cold
history backfills a tool frame's display from its linked approval so plan
blocks survive a refresh.
A completed background task delivers its notice into the conversation
as a user message with origin {kind: task}; the v2 stream dropped it,
so clients showed nothing. The projector now maps task.notified into
a user message carrying a structured notification payload — into the
running turn when busy, or the predicted delivery turn when idle —
and cold history rebuilds the same shape from turn.prompt/turn.steer
records by parsing the notification XML. The TaskNotified engine
event carries the source agent id so subagent completions stay
distinguishable.
A task notice delivered into a running turn persists as a context.append_message record, not turn.steer, so a refresh lost the notification user message. Cold history now surfaces task-origin appends (other origins stay hidden).
…tory Steered and task-notification user messages materialize at step boundaries, landing in the gap between one step's end and the next step's begin, where the strict per-step window silently dropped them. Timed users are now assigned to the first step sealed by their time, falling back to the last visible step.
The wire notification keeps the small structured fields and gains the raw XML envelope so clients render the card exactly like v1: clean body, output file chip, output preview, and the raw payload disclosure.
…ages
A skill activation is conversation content, not a system marker. The
system(skill) mark is gone; activations now surface as user messages
with origin {kind: skill}: model-tool and nested activations are
synthesized from skill.activated, user-slash injections arrive
through turn.steer, and prompts with bundled /skill mentions
populate skill_activations on the user message. Cold history mirrors
all three paths (skill turn origins are displayable, bundled
activations rebuild from turn.prompt).
prompt.submitted fires only for user-kind origins and carried no origin, so bundled /skill prompts lost their skill_activations and idle slash activations (skill_activation origin) produced a turn with no opening user message. The event now carries the origin, and the projector synthesizes the turn's opening user message from turn.started when no prompt message exists (cron/task fallbacks are deduped by the per-turn message count).
A model Skill tool call both fires skill.activated and steers the skill instructions into the turn, so synthesizing from the event duplicated the message. All activation paths are covered by the submitted/steered/turn-start messages, so the event emits nothing.
Model-tool and nested-skill activations are engine internals (v1 hides them); only user-slash activations surface as skill-origin user messages, in both the live projector and cold history.
A prompt with bundled /skill mentions persists the engineered skill blocks as leading content parts; cold history rendered them raw into the user bubble. The turn.prompt user text now slices them off like the engine's stripBundledSkillBlocks.
🦋 Changeset detectedLatest commit: d7f7f64 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d7f7f648ab
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+501
to
+508
| const wssV2 = registerWsV2({ | ||
| binder: v2Binder, | ||
| registry: connectionRegistry, | ||
| serverId, | ||
| sessionSourceFor: (sessionId) => liveSessionSourceFor(core, sessionId), | ||
| globalFanout, | ||
| logger: { warn: (meta, msg) => logger.warn(meta, msg) }, | ||
| }); |
There was a problem hiding this comment.
The new /api/v2/ws server is instantiated and accepted unconditionally, so every production process exposes the protocol even though this change keeps v1 as the current client surface. Register the kap-server flag with a default of false and guard both route registration and upgrade handling so the incomplete protocol is opt-in until release.
AGENTS.md reference: AGENTS.md:L64-L68
Useful? React with 👍 / 👎.
Comment on lines
+134
to
+139
| attach(source: V2SessionSource): SessionV2Binding { | ||
| let binding = this.bindings.get(source.sessionId); | ||
| if (!binding) { | ||
| binding = new SessionV2Binding(source, this.clock); | ||
| this.bindings.set(source.sessionId, binding); | ||
| } |
There was a problem hiding this comment.
After a live session is archived or closed and then restored in the same server process, attach returns the cached binding even though it is subscribed to the disposed session's event buses. SessionV2Binder.detach has no caller, so subsequent subscriptions to the restored session receive recovery from stale projector state and no new realtime events; compare the source generation or detach on session close before reusing this map entry.
Useful? React with 👍 / 👎.
Comment on lines
+185
to
+188
| private watchAgent(agent: V2AgentSource): void { | ||
| const permission = agent.permissionMode?.(); | ||
| if (permission !== undefined) this.projector.composer.apply({ permission }); | ||
| this.disposables.push(agent.bus.subscribe((event) => this.onAgentEvent(agent.agentId, event))); |
There was a problem hiding this comment.
When a REST caller starts a prompt before the first v2 subscriber connects, the binding is created after turn.started/turn.step.started; this forward-only bus subscription cannot replay those events, and no agent snapshot or pending-question synchronization is performed. Consequently recoveryFor is empty and current deltas are silently dropped until the next step because the projector has no currentStep; initialize the projector from live state/history before accepting the subscription.
Useful? React with 👍 / 👎.
Comment on lines
+198
to
+199
| const patch = factsPatchForEvent(event); | ||
| if (patch) this.emitFacts(patch, event.time ?? this.clock()); |
There was a problem hiding this comment.
When any subagent emits agent.activity.updated, agent.status.updated, goal.updated, or permission events, this path applies its patch to the single session composer and broadcasts it on the main session channel. A subagent turn can therefore replace the main agent's phase/model/permission in session.state; only main-agent facts should feed this composer, while the session work fields should continue to come from ISessionActivityView.
AGENTS.md reference: packages/kap-server/AGENTS.md:L33-L33
Useful? React with 👍 / 👎.
| const systems = out as { type: string; subtype: string; system_id: string }[]; | ||
| expect(systems.map((msg) => [msg.type, msg.subtype])).toEqual([ | ||
| ['system', 'hook'], | ||
| ['system', 'skill'], |
There was a problem hiding this comment.
This test always fails: AgentV2Projector.onSkillActivated deliberately emits nothing, so the constructed array contains five system messages while this assertion expects six and shifts every following entry. Since user-triggered skill activation is projected through the user-message/turn-steer path, remove the obsolete ['system', 'skill'] expectation and update the subsequent IDs.
Useful? React with 👍 / 👎.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
Internal protocol refactor — no linked issue. Client adoption: MoonshotAI/kimi-code-app#530. Continues the branch previously proposed as #3538 (fast-forward: same v2 stack plus the follow-up fixes below).
Problem
The v1 realtime channel carries two coexisting protocol families (legacy event frames and transcript frames), so one logical change has more than one source of truth. Clients cannot consume WS frames directly: every frame has to be applied through the transcript package before it becomes usable, and the frame granularity (off/turn/block/delta) is too low-level, forcing each client to rebuild message semantics on its own. Cold history is served in a different shape from the live stream, so refresh and reconnect recovery are ad hoc. Follow-up review against the real client exposed more gaps: approvals lost their display payloads, background task notices never surfaced, skill activations were a detached system marker, and several cold-history rebuilds silently dropped messages.
What changed
Adds a WS v2 protocol stack to kap-server, per
packages/kap-server/docs/2026-09-04-v2-protocol-refactor-plan.md:src/protocol/v2/— zod schemas for the full wire union: turn/step lifecycle, user/assistant/thinking/tool_call/system messages, task/plan/goal/sidechat/cron/attachment/compaction/undo frames, each carrying explicitstatustransitions. This is the only definition of the v2 wire shape; the code-app client re-exports it.src/services/v2Projection/projects engine loop events into wire messages —agentProjectorfor the live stream,coldHistoryfor a rebuild from persisted records. Parity-tested against 24 instance fixtures (test/v2Projection.test.ts).GET /sessions/{id}/historyspeaks the same wire shape as the live stream, paginates viabefore_turn, and reaches past compaction points./api/v2/wswith the hello/subscribe/ack handshake, omit-based unsubscribe, recovery payloads for refresh/reconnect, backpressure, and per-subagent channels.test/v2Engine.e2e.test.ts); v1 stays untouched and serves current clients.Follow-up fixes on top of the initial stack:
steerflag through prompt submission; the prompt route steers a pending handle within the same request), so the steered user message no longer renders twice.ToolInputDisplay, so an ExitPlanMode approval shows the plan review instead of an empty payload; cold history backfills a tool frame's display from its linked approval.task.notifiedprojects to a user message withorigin {kind: task}and a structured + raw notification payload — into the running turn when busy, or the task-origin delivery turn when idle. Cold history rebuilds both persistence forms (turn.prompt/turn.steer/context.append_message), and timed user messages survive step-boundary gaps.origin {kind: skill}instead of a detached system marker — only user-triggered (user-slash) activations surface, matching v1's hiding of model-tool activations. Bundled /skill prompts populateskill_activations; the turn's opening message is synthesized fromturn.startedwhen the prompt event is suppressed; cold history strips bundled skill blocks from the display text.before_turnpaging crosses compaction points.Checklist
gen-changesetsskill, or this PR needs no changeset (the v2 stack is an internal protocol change; the user-visible fixes each carry a changeset).gen-docsskill, or this PR needs no doc update (protocol decisions are recorded in the refactor plan doc in this branch).