Everything WebRTC carries should be timestamped at source

[pierre-rouanet]

Nothing this transport carries is timestamped at source. A frame arrives when it arrives, a robot.state notification arrives when it arrives, and a consumer that wants to know when the robot saw or felt something has only its own clock — which, over a relay, measures the network rather than the robot.

That is fine for driving a robot you are watching, and wrong for most of what a remote consumer is interesting for. SLAM is the case that makes it concrete: monocular SLAM on a stream with no capture times runs; the moment somebody wants visual-inertial — the IMU this robot has, at 50 Hz, on the same control channel — the two series cannot be related except by guessing.

Design note: remote-webrtc.md §11.

Two halves, and they are not the same problem

Media. RTP timestamps are relative to a random offset: they order frames and date none of them. The mechanism is the abs-capture-time RTP header extension, which carries a wall-clock capture time per packet. First job is checking what actually surfaces it — whether webrtcsink will negotiate it, and what a browser (RTCRtpReceiver.getSynchronizationSources, getStats) and aiortc expose of it. A header extension nothing on the receiving side surfaces buys nothing, and that determines whether this half is cheap or a fork of somebody's stack.

The control channel. Ours, and cheap: a monotonic reading plus the boot epoch that makes it comparable across processes, on every notification that describes a moment (robot.state first). The cost is a field per message and an argument about which clock — and the answer has to be the same clock the media path ends up dating frames with, or the two series still cannot be joined.

Why it is its own change

It alters what every notification looks like, so it wants its own API_VERSION bump and its own decision rather than riding along with a transport. Deliberately kept out of the remote-access slices (#209, #219).

Done looks like

• A consumer can take a decoded frame and a robot.state sample and say, with a bounded error it can state, which happened first.
• The error is bounded by something written down, not by the network.

[Nixxx19]

took the first job on the media half from source, since it decides the shape of the rest.

sender. webrtcsink does not know the extension. at the 0.15.3 that microduck-gst-plugins pins there is no abs-capture-time anywhere in gst-plugins-rs, and none at today's main either (cff4bf5), where what it negotiates is twcc and color-space. gstreamer core is the same: gst-plugins-good/gst/rtpmanager ships header extension elements for client-audio-level, mid, stream-id, repaired-stream-id, twcc and ntp-64, and nothing for abs-capture-time.

the nearest thing that exists is that ntp-64, urn:ietf:params:rtp-hdrext:ntp-64, a 64 bit ntp timestamp per packet that rtpfunnel already uses. abs-capture-time is the same 64 bit ntp value under a different uri, with an optional clock offset field after it. gstrtphdrext-ntp.c is 269 lines, so the sender side is an extension class in that shape and a line in webrtcsink to add it, not a fork of the stack. but no browser surfaces ntp-64, so on its own it only helps a receiver we write.

receivers. aiortc knows abs-send-time and not abs-capture-time (src/aiortc/rtp.py at 8a28646), so a python consumer gets nothing without a patch there as well. the browser i could not check from here. getSynchronizationSources() exposing captureTimestamp once the extension is negotiated is the thing to test live, and it is the one that decides whether the sender half is worth writing before a receiver exists.

so the sending side is small and ours, and the receiving side is one live browser check plus an aiortc patch. happy to do the sender half if that is the shape you want.