docs(spec): session JSONL streaming over MoQ (design, pre-implementation) - #697
docs(spec): session JSONL streaming over MoQ (design, pre-implementation)#697rsnodgrass wants to merge 2 commits into
Conversation
…ion) Co-Authored-By: SageOx <ox@sageox.ai> SageOx-Session: https://sageox.ai/repo/repo_019c5812-01e9-7b7d-b5b1-321c471c9777/sessions/2026-07-17T06-14-ryan-OxNBOE/view
📝 WalkthroughWalkthroughAdds a proposed specification for best-effort redacted session JSONL streaming over MoQ, covering identity, envelopes, lifecycle and cursor semantics, daemon and transport responsibilities, server projection, cloud contracts, redaction policy, and phased testing. ChangesSession streaming specification
Estimated code review effort: 2 (Simple) | ~15 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
…081/087) Co-Authored-By: SageOx <ox@sageox.ai> SageOx-Session: https://sageox.ai/repo/repo_019c5812-01e9-7b7d-b5b1-321c471c9777/sessions/2026-07-18T00-30-ryan-OxNBOE/view
|
Spec revised after a deep pass over the current conversation model (mono
Full mapping table ( |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/specs/session-streaming.md`:
- Around line 86-94: Update the session-streaming specification around the frame
kind and seq definitions to assign unique, collision-free identities to open,
lifecycle, and close frames. Reserve seq for turn entries and define a separate
frame ID with allocation, retry, and deduplication semantics, or explicitly
document a non-colliding sequence namespace and cursor behavior; ensure frame
retries cannot be dropped due to collisions with turns.
- Around line 172-178: Define an amendment mechanism compatible with
`(conversation_id, seq)` first-writer-wins deduplication: specify whether
amendments receive new sequence numbers, use an explicit revision/version field,
or follow an authorized replacement path. Update the finalization and late-frame
semantics around “Finalization is provisional and amendable; the FSM never
reopens” so `client_turn_id` provides idempotency without reusing rejected
sequence numbers, and ensure retroactive redaction cannot retain stale content.
- Around line 127-140: Define an atomic publication fence for pause activation
in the session publisher: persist the effective pause sequence as part of the
pause transition, then re-check that fence immediately before sending each turn.
If the turn falls at or after the effective pause sequence, suppress it and
advance the read cursor; ensure the fence and send decision are synchronized so
no suspended-range entry can leave the process.
- Around line 64-66: Update the cursor design and load/validation flow described
in the session streaming specification to detect when raw.jsonl has been
rewritten after recording stops. Persist a file generation or content identity
alongside the last sequence and byte offset, compare it against the current
raw.jsonl before resuming, and reset the cursor to the beginning on mismatch so
draining cannot skip or resume mid-record.
- Around line 86-91: Update the session-streaming specification’s lifecycle and
close-frame rules to define abort-specific wire behavior: specify the close
frame’s distinct finalization reason, ensure the server treats it as discarded
rather than normally stopped, and state whether buffered frames are dropped
before closure. Keep explicit-stop behavior unchanged and apply the rule
wherever abort is described.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 20cfffb7-828a-46d2-80a6-4635ef2d9cab
📒 Files selected for processing (1)
docs/specs/session-streaming.md
| Cursor (`stream-cursor.json`, session cache dir, 0600, atomic temp+rename) records the last | ||
| fully-published entry seq + byte offset. Safe to lose: restart from 0 is correct under | ||
| at-least-once. Clamped to file size on load. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Invalidate byte-offset cursors when raw.jsonl is rewritten.
Post-finalize draining is supported, but rewriteRawJSONL is only prohibited during active recording. A rewrite after stop can change line lengths or ordering while leaving the persisted byte offset apparently valid, causing the streamer to skip data or resume mid-record. Add a file generation/content identity to the cursor and reset it on mismatch, or block rewrites until draining completes.
Also applies to: 215-215
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/session-streaming.md` around lines 64 - 66, Update the cursor
design and load/validation flow described in the session streaming specification
to detect when raw.jsonl has been rewritten after recording stops. Persist a
file generation or content identity alongside the last sequence and byte offset,
compare it against the current raw.jsonl before resuming, and reset the cursor
to the beginning on mismatch so draining cannot skip or resume mid-record.
| - `kind`: `open` | `turn` | `lifecycle` | `close`. | ||
| - `open` — first frame: `session_id` (`ses_` twin), `session_name`, `repo_id`, `team_id`, | ||
| agent/coworker identity, adapter name (from the raw.jsonl header / `StoreMeta`). | ||
| - `turn` — one redacted conversation entry (mapping below). | ||
| - `lifecycle` — `{event: pause|resume|stop|abort, reason, suspended_from_seq?, resumed_at_seq?}`. | ||
| - `close` — final frame: `{finalize_reason: "explicit-stop"}` (see Finalization). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Define the wire behavior for abort.
The FSM includes abort and says it discards the session, but close only carries finalize_reason: "explicit-stop". If an aborted session emits that close frame, the server can finalize and project it as a normal stopped session. Add an abort-specific finalization reason/close rule, including whether buffered frames must be dropped.
Also applies to: 163-167
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/session-streaming.md` around lines 86 - 91, Update the
session-streaming specification’s lifecycle and close-frame rules to define
abort-specific wire behavior: specify the close frame’s distinct finalization
reason, ensure the server treats it as discarded rather than normally stopped,
and state whether buffered frames are dropped before closure. Keep explicit-stop
behavior unchanged and apply the rule wherever abort is described.
| - `kind`: `open` | `turn` | `lifecycle` | `close`. | ||
| - `open` — first frame: `session_id` (`ses_` twin), `session_name`, `repo_id`, `team_id`, | ||
| agent/coworker identity, adapter name (from the raw.jsonl header / `StoreMeta`). | ||
| - `turn` — one redacted conversation entry (mapping below). | ||
| - `lifecycle` — `{event: pause|resume|stop|abort, reason, suspended_from_seq?, resumed_at_seq?}`. | ||
| - `close` — final frame: `{finalize_reason: "explicit-stop"}` (see Finalization). | ||
| - `seq` — client-allocated, 1-based, monotonically increasing, **never renumbered** (stamped | ||
| into `raw.jsonl` at write time; see below). Sole ordering/dedup authority; server dedup key | ||
| is `(conversation_id, seq)`, first-writer-wins — duplicates are expected (at-least-once). |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Define unique identities for non-turn frames.
seq is the deduplication key, but the specification only defines its allocation for raw session entries. open, lifecycle, and close frames currently have no non-colliding sequence or frame ID; reusing a turn sequence would cause first-writer-wins drops, while omitting it makes retry/deduplication undefined. Reserve seq for turns and add a separate frame ID, or explicitly define a collision-free sequence namespace and cursor behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/session-streaming.md` around lines 86 - 94, Update the
session-streaming specification around the frame kind and seq definitions to
assign unique, collision-free identities to open, lifecycle, and close frames.
Reserve seq for turn entries and define a separate frame ID with allocation,
retry, and deduplication semantics, or explicitly document a non-colliding
sequence namespace and cursor behavior; ensure frame retries cannot be dropped
due to collisions with turns.
| ## Pause exclusion (publish-time, CRITICAL) | ||
|
|
||
| Pause/resume is upload-time filtering today (ADR-020): raw.jsonl keeps receiving entries | ||
| while paused. The session spine's rule is **fail-closed: paused means no content lands** | ||
| (off-the-record = `pause` + `reason:"off_the_record"`). The publisher therefore: | ||
|
|
||
| 1. Consults `RecordingState.SuspendedAt` + lifecycle pause markers per entry. | ||
| 2. **Never publishes** any `turn` whose seq lies in an open or closed suspended range — not | ||
| even a redacted tombstone (tombstones would still leak that something happened, and | ||
| fail-closed means nothing lands). The read cursor still advances. | ||
| 3. Publishes `lifecycle` frames instead: `pause` with `suspended_from_seq`, `resume` with | ||
| `resumed_at_seq`. The server marks `[suspended_from_seq, resumed_at_seq)` as | ||
| **intentionally absent** — not missing — so gap accounting (`missing_ranges`) never | ||
| requests a resend of withheld content. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🔴 Critical | 🏗️ Heavy lift
Add an atomic pause publication fence.
Because raw.jsonl continues receiving entries while the publisher reads and sends them, a turn can be read before pause activation but sent after it. The current wording does not define a linearization point that guarantees fail-closed behavior. Persist an effective pause sequence atomically and re-check it immediately before send, or otherwise fence publication so no entry in the suspended range can leave the process.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/session-streaming.md` around lines 127 - 140, Define an atomic
publication fence for pause activation in the session publisher: persist the
effective pause sequence as part of the pause transition, then re-check that
fence immediately before sending each turn. If the turn falls at or after the
effective pause sequence, suppress it and advance the read cursor; ensure the
fence and send decision are synchronized so no suspended-range entry can leave
the process.
| - **Finalization is provisional and amendable; the FSM never reopens** (ADR-087 D9). A | ||
| daemon that restarts after a `staleness-presumed` finalize keeps draining from its cursor: | ||
| the client state is `stopped` (control) × `draining` (durability debt) — two axes, not a | ||
| reopened session. Late frames are an amendment trigger: the server re-finalizes | ||
| idempotently (keyed on the delivered seq set), superseding derived artifacts with lineage. | ||
| Amendment MUST respect deletion tombstones — a purged conversation is never resurrected, | ||
| and the publisher drops its backlog on a tombstone response. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Make amendment semantics compatible with first-writer-wins deduplication.
A late amendment using an existing seq will be rejected by the stated (conversation_id, seq) first-writer-wins rule, so client_turn_id cannot provide the promised amendment idempotency. Define whether amendments use new sequence numbers, an explicit revision/version field, or an authorized replacement path; otherwise retroactive redaction and re-finalization can silently preserve stale content.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/specs/session-streaming.md` around lines 172 - 178, Define an amendment
mechanism compatible with `(conversation_id, seq)` first-writer-wins
deduplication: specify whether amendments receive new sequence numbers, use an
explicit revision/version field, or follow an authorized replacement path.
Update the finalization and late-frame semantics around “Finalization is
provisional and amendable; the FSM never reopens” so `client_turn_id` provides
idempotency without reusing rejected sequence numbers, and ensure retroactive
redaction cannot retain stale content.
1 participant
Summary
Lands the design spec for streaming coding-session JSONL to the cloud live over MoQ, in parallel with the existing stop-time git upload. Spec only — no code ships here. It exists so the format, transport, and safety contract get reviewed before the two implementation PRs (2a transport, 2b daemon) build against it, and so teammates and the mono side can start the cloud contract in parallel.
The design was hardened by four adversarial principal-engineer review panels; their confirmed findings are baked into the requirements (they are the reason this is worth reviewing as a doc first).
Why a spec PR now
internal/moqandsessionstreamare written against them.What the design commits to
raw.jsonlfrom a persisted cursor — no tee of the single-writer chokepoint; buffering is free and capture can't be blocked or corrupted by streaming.seq/eidstamping added toRawWriter— live entries carry none today; this is the stream's ordering/dedup key(session_id, seq).See the
## Data flowMermaid diagram and## Implementation phasestable in the doc for the shape and the PR split.Test Plan
docs/specs/session-streaming.mdrenders on GitHub (Mermaid diagram verified against the repo's strict-parse rules: quoted labels, no arrow substrings, no reserved IDs).Checklist (expand if needed)
/security-reviewon the diff, or N/A documented — N/A: no code; the security content of the design was produced by an adversarial review panel and is captured in the docSummary by CodeRabbit