Conversation
Adds per-line video to the intercom: WHIP publish and WHEP consume of H264 over SMB, video-source pinning with SSRC whitelisting, and the supporting session/line model fields (videoEnabled, hasVideo, isWhepReceiver, whepSourceSessionId) with pin reconciliation when a publisher leaves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bjowe
force-pushed
the
feat/video-support
branch
from
September 15, 2026 13:44
5846e58 to
c5398f1
Compare
The WHIP/WHEP answer preferred H264 whenever the client offered it, without ever consulting what the bridge can actually carry. SMB's compiled default is VP8, so any deployment that does not set codec.videoCodec advertises VP8 only -- while every browser, OBS and whip-mpegts offers H264. The publisher then encoded H264 that SMB could not forward, and every receiver got working audio with permanently black video and no error on any code path: the pin, the ssrc-whitelist and the keyframe request all succeeded, each side individually self-consistent. Select the most preferred codec present in BOTH the offer and SMB's advertised payload-types. Applied in configureEndpointForWhipWhep and createWhipWhepAnswer alike: these must agree, since the answer decides what the publisher encodes and the configure decides what SMB expects, and a divergence between them is precisely the silent failure. Reject explicitly when there is no overlap, naming both sides, and log the negotiation so a mismatch is visible. When SMB advertises no video payload-types the previous preference order is kept unchanged, so existing deployments are unaffected. Verified on the OSC review env: a browser WHIP publisher against the VP8-only catalog SMB reported outbound-rtp codec video/H264 while a pinned consumer received no inbound video stream at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hasVideo advertises a session as a pin source. The WHIP path set it from offerHasVideo -- the mere presence of a video m-line -- while the SSRCs it depends on were stored only conditionally, when the offer's FID group or a usable a=ssrc actually parsed. An offer with a video m-line but no parseable SSRCs therefore persisted hasVideo:true with an empty video.ssrcs. The publisher showed up as pinnable in the UI, but every pin to it resolved to an empty ssrc-whitelist and 425ed forever; the frontend retries 425 a few times, gives up silently, and leaves the tile on the previously pinned source. The browser path already guards against this (api_productions.ts) and its comment claims the WHIP path uses the same rule -- it did not. Bind the flip to the SSRCs actually persisted, and warn when an offer carries video that cannot be advertised, which was previously silent and left no way to recover the offer's shape from the logs afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Match the comment style established in "fix: clean up": short and factual about what the code does, rather than multi-line rationale blocks. Comments only -- no behaviour change, and the tests are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
birme
requested changes
Sep 17, 2026
birme
left a comment
birme
left a comment
Contributor
There was a problem hiding this comment.
Automated code-reviewer verdict (daily-backlog-pr Phase 3): NEEDS CHANGES
Reviewed the full +2611/−191 diff across 24 files. Substantial, mostly careful WebRTC/SDP work with strong test coverage in several areas — but two real defects should block:
Blocking
src/api_productions.ts(~:1126) —reply.code(204)lost its.send(). The diff changesreply.code(204).send()→reply.code(204)on thePATCH /session/:sessionIdhandler (the browser's WebRTC handshake finalize call). The handler isasyncand now neither calls.send()nor returns a payload, so Fastify never flushes the response and the request hangs until socket timeout. Critical connect path, no test covers it. Restore.send()(and note 204 isn't in the declared response schema — 200/400/500 only).
High
- WHIP publisher DELETE does not reconcile pinned receivers (
src/api_whip.ts~:328). The/session/:sessionIdDELETE correctly doesgetReceiversPinnedToSession→reconfigureEndpoint(strip stalessrc-whitelist) →updateSessionVideoPin(..., null), but the WHIP DELETE path (how a publisher actually leaves) only callsclearWhepSourceIfPinnedand skips the receiver-whitelist reconciliation. Receivers pinned to the departed publisher keep a stale whitelist → frozen video until renegotiation. The headline "pin reconciliation when a publisher leaves" feature is only wired into one of the two publisher-removal paths.
Medium
- Unbounded
pinnedSessionIdstrings on newSetLineWhepSourceRequest/SetSessionVideoSourceRequestschemas (nomaxLength) — consistency/known-concern class here (low exploitability: used only as CouchDB_idlookup). VideoSmbPayloadParameterswidened toType.Record(Type.String(), Type.String())— removes key validation; add a comment / value-length bound.GET /session/:sessionId/namehas noschemablock at all (params or response) — inconsistent with every other route; new unauthenticated read surface.
Nits
- No tests for the 204 answer path (would have caught the blocker),
PATCH .../whep-source, orGET .../name. anytyping whereSfuVideoStreamexists; positional-boolean growth inallocateEndpoint/createEndpoint.
Good: correct ISmbProtocol usage, the transport-media find-by-fingerprint fix, keyframe cycle, NaN feedback guard, and codec-overlap rejection are solid and tested. Fix the blocker + WHIP-delete gap and this is close.
…P delete (Eyevinn#341) Restore reply.code(204).send() (and the 204 schema) on PATCH /session/:sessionId so Fastify flushes the WebRTC handshake finalize response instead of hanging until socket timeout. Mirror the /session/:sessionId DELETE receiver-whitelist reconciliation into the WHIP publisher DELETE path so receivers pinned to a departed publisher have their stale ssrc-whitelist stripped instead of freezing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds per-line video to the intercom: WHIP publish and WHEP consume of H264/VP8 over SMB, video-source pinning with SSRC whitelisting, and the supporting session/line model fields (videoEnabled, hasVideo, isWhepReceiver, whepSourceSessionId) with pin reconciliation when a publisher leaves.
Closes #341
Companion PR (frontend): Eyevinn/intercom-frontend#684