feat(protocol): define realtime synchronization protocol - #2
Merged
Conversation
Freeze the transport-agnostic Sitrep realtime protocol (v1.0.0) covering all 13 message types: hello, resume, snapshot, delta, ack, task.event, message.event, metric.frame, subscribe, unsubscribe, interest.renew, command, and error. SPEC.md pins down the semantics three independent teams (TS server, Go Agent, Swift app) need to implement without further clarification: reliable-event delivery via per-device device_seq and (device_id, device_seq) dedup; space_revision accounting with the resume->delta/ snapshot decision flow; a 30-60s interest lease with server-driven throttle/resume_rate notification to sources; viewer-initiated commands (pause/resume/stop/run_now) with idempotency keys and TTLs; hello-based version negotiation; a strict envelope/credential boundary and a per-message-type authorization matrix; integer-Unix-millisecond timestamps everywhere, with a schema-level bound that rejects seconds-scale values; plain-text ping/pong heartbeats decoupled from protocol state; a full error-code table; and a server cost table. JSON Schemas (draft 2020-12) live in common.schema.json, envelope.schema.json, and one file per message type under messages/, composed via $ref rather than duplicated.
Add at least one valid fixture per message type, one invalid fixture per constraint SPEC.md calls out as rejectable (missing device_seq, a seconds-scale timestamp, an oversized metric value, a viewer attempting a server-only command action, a credential leaking into the envelope, and more), and five end-to-end scenario sequences with a README each: agent-reconnect-replay, client-resume-delta, client-revision-gap-snapshot, interest-lease-expiry, and duplicate-device-seq. These are shared test data for the TS server, Go Agent, and Swift app implementations.
Add a self-contained Node script (own package.json/package-lock.json, pinned ajv@8.20.0, zero dependency on or from server/, daemon/, or apple/) that loads every schema in proto/realtime/ into one Ajv (draft 2020-12) instance and asserts every valid fixture and scenario message validates while every invalid fixture is rejected. `npm install && npm run validate` inside proto/realtime/tools reports 57/57 fixtures with the expected result.
Adopt the protocol owner's rulings from the adversarial review, still within the unreleased 1.0.0: - delta becomes the sole viewer-facing reliable-event carrier: task.event and message.event are uplink-only, and every applied reliable event is re-emitted as a broadcast single-event delta carrying its revision; mandatory viewer sequence hello -> subscribe -> resume with exact from_revision apply/discard/gap rules (F1) - resume always yields exactly one reply: snapshot for last_revision 0 or an unservable gap, an explicit empty delta when already current; all "MAY send nothing" wording removed (F2) - snapshot chunking via part/final sharing one revision, and chained catch-up deltas split on event boundaries; framing section lists every split mechanism (F3) - command gains task_id/automation_id with a per-action required-field matrix aligned to the HTTP control surface; params reserved with no v1 keys (F4) - new 14th type config.event: server-minted reliable event with no device_id/device_seq, identity = its revision slot, carried inside delta; snapshot gains automations; matrix and cost table updated (F5) - envelope top level permanently strict (unknown top-level field is malformed forever); body-only unknown-field tolerance (F6) - handshake hardening: nothing between offer and accept, pre-accept frames get hello_required + close, 10 s recommended timeouts, retryable-means-after-reconnect semantics (F7) - interest leases decoupled from connections: per device, ended only by expiry or unsubscribe, lazy expiry allowed, notifications only on 1<->0 edges; interest.renew wholesale-replaces topics; unsubscribe acked via in_reply_to; only the metric topic filters in v1 (F8, F11, F14) - command relay preserves original ts and command_id, TTL checked at relay and re-checked at the source with +/-30 s skew; client-sent origin "server" is always unauthorized (F9) - uplink events with a foreign body.device_id rejected as unauthorized (F10); device_seq scope pinned to (device, space) (F15) - snapshot.metrics documented as best-effort ephemeral cache; snapshot only ever replies to resume (F12, F17) - new deterministic event-folding section: field-by-field task state folding, config folding, and normative message-window truncation with the delta/snapshot history asymmetry declared acceptable (F13) - ack.acked is an exact enumeration scoped to the receiving device (F14) - connection supersession: newest connection wins, old one closed with new error code superseded; routing follows the newest connection; no duplicate lease counting (F16) - nits: metric_sample gains target/min/max/alert_above/alert_below, message.event composes the shared reliability $def, hello directions annotated offer/accept, internal_error must not leak internals
- new valid fixtures: config.event, the empty you-are-current delta, and the superseded error; snapshot fixture gains part/final/automations and metric geometry; delta fixture exercises the config.event branch; command pause fixture carries its now-required task_id - new invalid fixture role-viewer-command-origin-server.json using the sender_role wrapper form, guarding the rule that a client-sent command with origin "server" is unauthorized even though the frame itself is well-formed - client-resume-delta and client-revision-gap-snapshot scenarios follow the mandatory hello -> subscribe -> resume order; resume-delta ends with a live single-event delta; revision-gap shows a two-chunk snapshot (part 1 + final part 2, same revision) - new duplicate-connection-supersede scenario: same device opens a second connection, server accepts it and closes the first with superseded - interest-lease-expiry README reworded for per-device leases decoupled from connections, lazy expiry, and 1<->0 edge-only notifications
Mirror the SPEC section 10.1 client-send matrix (including the rule that
a client may only send command with origin "viewer") in validate.js, and
accept an optional {sender_role, frame} wrapper form for fixtures whose
invalidity is role-dependent rather than shape-dependent. Bare-envelope
fixtures keep their pure schema check.
Final pre-freeze rulings from the second review round:
- delta eligibility is per connection: every connection, including a
superseding reconnect, must complete hello -> subscribe -> resume in
order before receiving any delta; resume is now required per
connection. The server sends no live delta on a connection until it
has sent that connection's resume reply - always the first
delta-family envelope - and viewers defensively discard any pre-reply
delta, justified by in-order frame delivery (the reply necessarily
covers such a delta's content). This removes the cold-reconnect window
where a live delta could race an uninitialized local revision and
trigger spurious resumes (NF1, NF3)
- lease survival across supersession affects only throttle-edge
accounting: supersession never emits throttle/resume_rate and never
confers delta eligibility; the local revision is re-established from
the new connection's resume reply with no cross-connection
inheritance (NF1d, NF3)
- while a chunked snapshot is in flight the server defers every other
outbound envelope on that connection (including acks) until after the
final chunk; only ping/pong may interleave, and a viewer treats any
other interleaved envelope as a malformed sequence (NF2)
- clients may only send hello{stage: offer}; a client-sent accept is a
handshake violation answered with hello_required and a close (NF4)
- config.event minting and its revision increment form a single durable
transaction, with duplicate-resistance equivalent to the
(device_id, device_seq) dedup of uplinked events; HTTP control-plane
retries cannot mint duplicates (NF5)
- command object-field exclusivity enforced by schema: pause/resume/stop
forbid automation_id, run_now forbids task_id, throttle/resume_rate
forbid both; two invalid fixtures added (NF6)
- unified free-text caps via common $defs (free_text <= 2048 chars for
title/step/message/text, label_text <= 256 for labels/names) so a
single record can never exceed the 64 KiB frame limit, stated in the
framing section (E1)
- authorization layer gains the client-may-only-send-hello-offer rule, with the role-client-hello-accept invalid fixture exercising it (NF4) - delta revision arithmetic (to - from == events.length) checked on every delta fixture, since JSON Schema cannot express it - per scenario directory, in filename order: consecutive deltas must chain from/to exactly; snapshot chunk runs must share one revision, number parts consecutively from 1, tolerate no interleaved envelope, and close with final: true; device_seq must be non-decreasing per device (equal values allowed as retransmissions), covering both uplink bodies and delta-embedded events (E2) - still zero dependencies beyond the pinned ajv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Risk