feat(mediajson): add stop event and talk timestamps on start/stop - #239
Conversation
Add the mediajson `stop` event (StopEvent/Stop), mirroring the VoxImplant
protobuf StopEvent (`tag` + optional MediaInfo{bytesSent, duration}), and
register it in @JsonSubTypes so peers can parse it.
Augment Start and Stop with an optional RTP `timestamp` (media clock, e.g.
48000 Hz for Opus; string-encoded like Media.timestamp) so a peer can bracket
a contiguous run of media (a "talk") on the media timeline: start = the first
packet's timestamp, stop = one past the last packet. Omitted for a plain
start/stop. Link the VoxImplant protobuf definition in the file header.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #239 +/- ##
============================================
+ Coverage 34.70% 35.40% +0.70%
Complexity 162 162
============================================
Files 36 36
Lines 1383 1398 +15
Branches 140 140
============================================
+ Hits 480 495 +15
Misses 870 870
Partials 33 33
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
| * run of media (a "talk"), when a peer uses start/stop to bracket one (paired with a [StopEvent]). Null when the | ||
| * start event only announces the media format. Encoded as a string like [Media.timestamp]. | ||
| */ | ||
| @JsonSerialize(using = Long2StringSerializer::class) |
There was a problem hiding this comment.
Serializing numbers as strings was only necessary to keep the format the same as the original. Suggest encoding as JSON numbers since this is an addition anyway.
There was a problem hiding this comment.
Done in 5f7c6bf — Start.timestamp and Stop.timestamp are now plain JSON numbers (dropped the Long2StringSerializer/String2LongDeserializer), matching Media.audioLevel and what the proxy already emits on the wire. I left MediaInfo (bytesSent/duration) string-encoded, since it's part of VoxImplant's original stop format rather than a new addition.
| val bytesSent: Long, | ||
| @JsonSerialize(using = Long2StringSerializer::class) | ||
| @JsonDeserialize(using = String2LongDeserializer::class) | ||
| val duration: Long |
There was a problem hiding this comment.
Clarified in the KDoc (5f7c6bf): duration is in milliseconds, whereas Start.timestamp/Stop.timestamp are on the RTP media clock (e.g. 48000 Hz), not wall-clock ms — I called out the unit difference explicitly so the mismatch is intentional and documented.
Per review feedback: Start.timestamp and Stop.timestamp are new augmentations, not part of VoxImplant's original format, so the string encoding (only needed for VoxImplant format-fidelity) is unnecessary. Drop the Long2StringSerializer/String2LongDeserializer and emit them as natural JSON numbers -- matching Media.audioLevel and what the proxy already sends on the wire. MediaInfo (bytesSent/duration) is part of VoxImplant's original stop format, so it deliberately stays string-encoded. Also clarify units: duration is in milliseconds, whereas the timestamp fields are on the RTP media clock (e.g. 48000 Hz). Tests assert the encoding explicitly (isNumber / isTextual) and keep coverage that a string-encoded timestamp is still leniently parsed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 participants
Summary
Adds the mediajson pieces to bracket runs of translated/synthetic audio ("talks") with start/stop events carrying RTP timestamps.
stopevent (StopEvent/Stop), mirroring VoxImplant's protobufStopEvent(tag+ optionalMediaInfo{bytesSent, duration}), registered in@JsonSubTypesso peers can parse it.StartandStopgain an optionaltimestamp(RTP media clock, e.g. 48 kHz for Opus; string-encoded likeMedia.timestamp) marking the first packet of a talk and one-past-its-end. Omitted for a plain start/stop, so existing senders are unchanged.Testing
mvn -pl jicoco-mediajson test(Kotest): serialize/parse for start-with-timestamp, stop with/without timestamp, stop withmediaInfo, and number/string numeric encodings.mvn -pl jicoco-mediajson ktlint:checkclean.Related
Consumed by the paired jitsi-videobridge and opus-transcriber-proxy changes for the audio-translation "sending" indicators (client side: lib-jitsi-meet #3066).
🤖 Generated with Claude Code