Add a SourcesEvent to MediaJson - #235
Merged
JonathanLennox merged 1 commit intoJun 12, 2026
Merged
Conversation
Add a bridge-sent "sources" event carrying the source names the bridge will
export to the peer and the source names it requests from the peer:
{"event":"sources","exports":[...],"requests":[...]}
Like SessionEndEvent it is only ever sent by the bridge, so it is not
registered in Event's @JsonSubTypes parse path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #235 +/- ##
============================================
+ Coverage 34.08% 34.27% +0.19%
Complexity 162 162
============================================
Files 36 36
Lines 1370 1374 +4
Branches 140 140
============================================
+ Hits 467 471 +4
Misses 870 870
Partials 33 33
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
jitsi-ci Bot
pushed a commit
to jitsi/jitsi-pr-tests-pages
that referenced
this pull request
Jun 11, 2026
bgrozev
approved these changes
Jun 12, 2026
JonathanLennox
added a commit
to JonathanLennox/jitsi-videobridge
that referenced
this pull request
Jun 12, 2026
Picks up the SourcesEvent / inbound MediaEvent handling from jitsi/jicoco#235 that this branch needs, so it builds against a released jicoco instead of a local SNAPSHOT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bgrozev
added a commit
to jitsi/jitsi-videobridge
that referenced
this pull request
Jul 6, 2026
…sources, inbound injection (#2419) * feat: Plumb the synthetic flag and connect exports/requests/type Carry the new jitsi-xmpp-extensions parameters through the videobridge model without acting on them yet: - MediaSource.isSynthetic() flows into MediaSourceDesc/AudioSourceDesc via MediaSourceFactory and Colibri2ConferenceHandler. - Connect's exports/requests source-name lists and type are read in ExporterWrapper and stored on the Exporter (surfaced in debugState). Also bump the jitsi-xmpp-extensions dependency to 1.0-113-g4281255 to pick up these new elements. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Support multiple connects in colibri2 ExporterWrapper now manages one Exporter per connect instead of rejecting more than one. The same audio is fanned out to every exporter; each is handed its own clone of the packet (the last gets the original) since each exporter takes ownership of its buffer. If any connect is rejected, the already-started exporters are stopped so none are left behind. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Reconcile connects by URL instead of rejecting changes Allow the set of connects to change while running, using the connect URL as its identity: - stop exporters whose URL is no longer requested, - start exporters for URLs that weren't already running, - for a URL that is still requested but whose other parameters changed, pass the update to the existing exporter (Exporter.update() is a placeholder for now; applying the change is a follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Apply connect updates for changed exports/requests, reject others Implement ExporterWrapper's previously-placeholder update path. When a still-running connect (matched by URL) changes only its exported/requested source names, apply them to the live Exporter via Exporter.update(). Any other change (HTTP headers, ping, type, audio) is connection-level and can't be applied in place, so it's rejected up front with FeatureNotImplementedException before any exporter is stopped or started. Add an injectable Exporter factory (via an internal constructor; the public constructor is unchanged) so the reconcile, update and error paths can be unit-tested against a mock Exporter, and add ExporterWrapperTest covering start, stop, reconcile, in-place update, and rejection of non-updatable changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Don't route synthetic audio sources except on explicit subscription Synthetic audio sources (e.g. bridge-generated translated audio) are now: - not routed to a client automatically, even under an "All"/"Exclude"/default subscription -- only to an endpoint that explicitly (Include) subscribed to them by name (AudioSubscriptionManager + AudioSubscription.isExplicitlyWanted); - excluded from speech-activity / loudest-speaker selection, and never dropped by loudest-only filtering (Conference.levelChanged, now keyed by the source's SSRC instead of assuming a single source per endpoint); - still forwarded to relays unchanged. AbstractEndpoint indexes its audio sources by SSRC for an O(1) per-packet lookup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Let audio subscriptions resolve relayed sources Relayed (remote) endpoint audio sources now participate in the conference's AudioSubscriptionManager the same way local sources do: RelayedEndpoint feeds add/removeAudioSources from its audioSources setter and on expiry, and Conference.getAudioSourceDescs() now also includes relayed sources (via a new Relay.getAudioSources()). This makes audio subscriptions work for relayed sources: - Include subscriptions can now resolve a relayed source's name to its SSRC; - Exclude subscriptions now deliver relayed audio (previously wantedSsrcs was built only from local non-excluded sources, so an Exclude subscriber received no relayed audio at all); - relayed synthetic sources are now suppressed for "All" subscriptions on a receiving bridge, matching the behavior for local synthetic sources. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Send sources event, filter exports, expose exporters as packet handlers Implement the outbound side of the exports/requests connect feature on the media-json exporter: - On connect (and on update) the exporter sends a media-json "sources" event declaring its exported/requested source names to the peer. - Outbound audio is filtered by the connect's exports: an empty/absent exports list means "export all audio" (the existing recorder/transcriber behavior), while a non-empty list forwards only audio from the named sources. (Absent and empty are indistinguishable in colibri2, so both mean "all".) - Each Exporter is now a PotentialPacketHandler in its own right; ExporterWrapper exposes them via getPacketHandlers() and Conference.sendOut() folds them into its existing clone-minimizing handler loop. This removes the previous double-cloning (conference -> wrapper -> per-exporter) and avoids cloning a packet for an exporter that filters it out. Requires the SourcesEvent added in jicoco (jitsi/jicoco#235); builds against jicoco 1.1-SNAPSHOT until that is released. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Inject received translated audio as its synthetic source Implement the inbound side of the translator connect: translated audio that the translator returns over the media-json websocket is injected into the conference as the corresponding synthetic source. - Exporter dispatches incoming media events via a new handleMediaEvent callback (with a media_events_received metric); ExporterWrapper threads it through. - Conference.handleMediaMessage maps the event's tag to the synthetic AudioSourceDesc of that name, resolves the conference-negotiated Opus payload type (Endpoint.getOpusPayloadType; payload types are never rewritten, so it's consistent across endpoints), builds an Opus RTP packet on that source's SSRC, and injects it via handleIncomingPacket. The synthetic-source routing then delivers it to explicit subscribers, keeps it out of loudest-speaker, and forwards it to relays. - sendOut no longer echo-suppresses synthetic-source packets: such audio is bridge-generated, not an endpoint's own live audio, so an endpoint (the source's owner) can subscribe to and receive its own synthetic source. This holds whether the audio was injected locally or received over a relay (where the relay attributes the packet to the owner endpoint). Requires the SourcesEvent/MediaEvent handling in jicoco (jitsi/jicoco#235); builds against jicoco 1.1-SNAPSHOT until that is released. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Bump jicoco.version to 1.1-172-g652baf4 Picks up the SourcesEvent / inbound MediaEvent handling from jitsi/jicoco#235 that this branch needs, so it builds against a released jicoco instead of a local SNAPSHOT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Tag exported media-json by source name The exporter now tags outbound start/media events with the source name (the same identifier used by exports/requests and the sources event) instead of "<endpointId>-<ssrc>", so the whole media-json protocol keys on source names end to end. Falls back to the old endpoint+ssrc form only when a source name isn't known for the SSRC. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Log injected synthetic source creation; handle post-unsubscribe media race - Log (once) when an injected synthetic source is first created from received translated media. - The unknown-tag warning is now logged once per tag rather than per packet. - Media that arrives for a synthetic source just after it was unsubscribed (the translator sent it before seeing the unsubscribe) is an expected transient: it is dropped at debug level instead of warning about an unknown source. Tracked via removedSyntheticSources, cleared when the source is re-added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: Use all/include/exclude lists for ReceiverAudioSubscription The previous ReceiverAudioSubscriptionMessage had an exclusive mode (All/None/Include/Exclude) with a single list whose meaning depended on the mode. That couldn't express "all regular audio plus these specific synthetic sources", which is needed now that synthetic sources are only delivered on an explicit subscription. Replace it with a data class carrying always-present fields: - all: default for any source not named in include/exclude (synthetic sources excepted -- they require an explicit include) - include: always received, in addition to whatever `all` selects; the only way to receive a synthetic source - exclude: never received; takes precedence over all and include AudioSubscription/AudioSubscriptionManager are updated accordingly (isSsrcWanted: exclude -> include -> all; isExplicitlyWanted gates synthetic sources on include only). A @JsonCreator factory parses both the new form and the legacy mode/list form, translating the latter into the new semantics so no code outside the parser is aware of the old syntax: All -> all=true; None -> all=false; Include -> include=list; Exclude -> all=true, exclude=list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ref: Apply colibri2 connects as id-keyed create/expire deltas (#3) * ref: Apply colibri2 connects as id-keyed create/expire deltas. ExporterWrapper now keys exporters by connect id instead of URL and applies the signaled connects as a delta: a connect marked create starts a new exporter (throwing if the id already exists), expire removes one (throwing on unknown id for create, or on a non-create change to an unknown id), and an unmarked connect updates the existing one. Connects not mentioned in a request are left untouched. This makes message-ordering problems visible instead of silently mishandled. * chore: Bump jitsi-xmpp-extensions to 1.0-114-g087129a. * fix: Resolve audio subscriptions and synthetic routing against known sources Make AudioSubscriptionManager the single, lock-consistent source of truth for the conference's known audio sources, closing several races where audio was misrouted around the moment a source was signaled or removed via colibri. - Subscriptions now resolve include/exclude names against the manager's own knownSources set (maintained under the same lock as subscription updates), not a snapshot the caller gathered outside the lock. Previously a ReceiverAudioSubscription arriving concurrently with a source's colibri signaling could resolve against a stale snapshot and permanently drop the source. setEndpointAudioSubscription no longer takes a source list. - findSyntheticAudioSource (used to inject translated media) now resolves via the manager rather than scanning each endpoint's audioSources field, which is updated out of step with the manager (local endpoints update the manager before their field, relayed endpoints after) and could miss a just-added synthetic source. - isEndpointAudioWanted gates routing on a single immutable classification snapshot (ssrc -> isSynthetic; absent => unknown): an SSRC that corresponds to no currently-known source is no longer routed under the permissive default. This closes the synthetic-source leak in both directions -- media for a synthetic SSRC not yet known (early) or just removed (late) was otherwise forwarded to non-subscribers as ordinary audio. Injected synthetic packets bypass the routeLoudestOnly energy filter entirely, so this had to be enforced at the routing decision itself. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Expose audio subscription and synthetic source debug state Add debugState() to AudioSubscription and AudioSubscriptionManager and include it in Conference.getDebugState (non-SHORT modes) under "audio_subscriptions", so the conference /debug output now shows this branch's audio-routing state: - known_sources: every known audio source (ssrc, name, owner, synthetic flag) -- the authoritative set that routing and name resolution are gated on. - subscribed_local_sources: source name -> the endpoints with an Include subscription to it. - subscriptions: per-endpoint mode (all) plus include/exclude names and the SSRCs they currently resolve to (included_ssrcs/excluded_ssrcs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: Tag each exporter's debug info with its connect id ExporterWrapper.debugState() listed exporters as an anonymous array, so debug entries couldn't be correlated to a specific connect. Add a "tag" field (the connect id each exporter is keyed and signaled under) to each array item. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: Cap exporter reconnect backoff instead of overflowing to a negative delay getDelayMs applied toLong() to 2.0.pow(attempt - 2) before taking the min with maxDelay. For a large attempt count the pow overflows to Infinity, toLong() yields Long.MAX_VALUE, and baseDelay * Long.MAX_VALUE overflows Long to a negative value, so min(negative, maxDelay) returned a negative delay and the exporter reconnected with no backoff (tight loop). Take the min in Double space and convert afterwards so it clamps to maxDelay. * refactor: Remove dead audio-source relay-merge path getAudioSourceDescs() had no callers and Relay.getAudioSources() was used only by it. Relayed audio sources are registered via the RelayedEndpoint.audioSources setter calling conference.addAudioSources(), and subscriptions resolve against AudioSubscriptionManager's known sources, which is the authoritative set. Remove both unused methods and fix the now-dangling reference in the findSyntheticAudioSource comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: Use a concurrent map for exporters to avoid CME in debugState The exporters map is mutated on the colibri thread but iterated on the debug thread by debugState(), which could throw ConcurrentModificationException. * fix: Fall back to a scan when the per-SSRC audio source index is stale levelChanged() resolves the source by SSRC via the volatile index, which is updated just after the endpoint's audioSources list. A level packet arriving in that window found no source, skipping the synthetic-source exemption and the explicit-subscription guard, so the packet could be dropped by loudest-only filtering. Fall back to scanning the endpoint's audio sources by SSRC. * fix: Do not export synthetic audio to exporters An exporter with an empty exports list wants all audio, which would include bridge-generated synthetic (translated) audio -- feeding a translator its own output, or sending injected audio to a transcriber. Skip all exporters for synthetic packets. * fix: Bound the synthetic-source log-suppression sets loggedUnknownMediaTags and removedSyntheticSources could grow for the lifetime of the conference: unknown media tags are never removed, and removed synthetic sources are only cleared when a source with the same name is re-added. Bound both with an LRU set; they only gate logging, so eviction at worst produces one extra log line. * fix: Keep the mediajson tag consistent and only export known sources The mediajson tag was recomputed per packet while the start event fires once per SSRC, so a source name that only became resolvable mid-stream flipped subsequent media events to a tag no start event had announced. Resolve the tag once when the SSRC's stream state is created and store it there; on a payload type change the new state (announced by a new start event) keeps the tag, since an SSRC's source name doesn't change. The unresolvable case is now dropped rather than falling back to an "epId-ssrc" placeholder tag: source names are mandatory in colibri2, so an SSRC with no known source is media racing colibri signaling, and such audio isn't routed to any receiver -- so it shouldn't be exported either. Exporter.wants now requires the source name to resolve even in export-all (empty exports) mode, mirroring receiver routing. Also, on a payload type change the current packet was encoded with the stale pre-change state (old clock-rate offset and trackers) even though the replacement state had already been stored, producing one media event discontinuous with both neighbors; encode it with the new state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: Validate same-id connect combinations within one delta Pass-1 validation only consulted the live exporter map, so within a single delta: two creates for the same id both passed and the second silently overwrote the first in pass 2, leaking the first (started) exporter until conference expiry; an expire+create of the same id (the natural way to replace a connect's URL, since URL changes are rejected as non-updatable) was wrongly rejected as already-existing; and an expire+update of the same id passed validation but crashed pass 2 with an uncaught NoSuchElementException after the expire removed the entry. Collect the delta's expires first, then check creates against the map as it will be after they apply: expire+create of one id now replaces it (in either order within the delta), and any other repeated mention of an id is rejected up front, before anything mutates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: Synchronize exporter connect application and stop The concurrent map alone made individual operations safe but not the compound ones: applyConnects (on the conference's colibri queue, or on a Jetty thread for colibri-over-REST) validates against the map before mutating it, and could interleave with stop() on the conference-expiry thread, starting an exporter into the cleared map where nothing would ever stop it (a leaked websocket and ping timer). Synchronize both methods, and have applyConnects reject deltas after stop() has run. debugState still iterates the concurrent map without the lock, and the media path still reads the volatile packet-handler snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ref: Build injected packets from the buffer pool with standard headroom Injected translated-audio packets allocated a fresh exact-fit array per packet with no room before the header, unlike every other packet-construction site: downstream returnBuffer calls couldn't pool the odd-sized arrays, and adding the abs-send-time header extension on egress forced a reallocation and copy for each packet. Get the buffer from ByteBufferPool with the standard head and tail room instead, and zero the header region since pooled buffers aren't cleared. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ref: Cache the conference Opus payload type; warn once when missing getOpusPayloadType() scanned every local endpoint on each injected media event (~50/sec per synthetic stream) to recompute a value that is conference-constant, and the no-payload-type warning fired unconditionally at the same rate, unlike the deduped logging of the adjacent unknown-tag path. Cache the first resolved payload type and log the warning once per conference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: bgrozev <boris@jitsi.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a bridge-sent
sourcesevent to the MediaJson protocol, carrying the source names the bridge will export (send out) to the peer and the source names it requests (wants to receive) from the peer:{"event":"sources","exports":["523834112-a0","2394a3432-a0"],"requests":["523834112-a0.en","2394a3432-a0.hi"]}This will be used by the videobridge exporter to tell a translator/transcriber connection which sources it is sending and which (e.g. translated) sources it wants back, corresponding to the new
exports/requestschild elements on the colibri2<connect>element.Like
SessionEndEvent, this event is only ever sent by the bridge, so it is not registered inEvent's@JsonSubTypesparse path.Includes
MediaJsonTestcoverage for serialization (populated and empty lists).🤖 Generated with Claude Code