From 18052dbb98940c2f23a48aceb4752a33dc43ef3c Mon Sep 17 00:00:00 2001 From: Quchaosheng Date: Tue, 15 Sep 2026 15:06:36 +0800 Subject: [PATCH] The clock bridge gets a reader, and states what its bound is worth MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit robot.state, tof.frame and head_imu.frame already carry t_ns, and media.video already publishes the mono_ns/real_ns pair (v24). Nothing consumed the pair: the bridge was on the wire with no reader. proto::clock::ClockPair is that bridge as a type. media.video's params now read their pair through ClockPair::now rather than as two bare adjacent calls, which puts "these two readings belong together" in the type instead of in a comment above them and leaves no way for the next caller to read them a second apart. ClockPair::wall is the receiving end of the same contract — the addition that turns a t_ns into a wall-clock moment. It has no caller in this workspace, and that is stated rather than glossed: the receivers that need it are the ones §11 describes, a peer that reads RTP rather than a robot daemon. The offset is signed, because a sample can predate the pair it is converted with, and the result is clamped rather than allowed to wrap. The doc comment is where the bound §11 asked for lives, and getting it right took two attempts. The first claimed Linux slews rather than steps NTP adjustments, which is false of CLOCK_REALTIME and false of this hardware in particular: the board has no battery-backed RTC, boots at 1970, and NTP steps the clock during provisioning (scripts/provision.sh says so; pad-link-test.sh refuses the wall clock for the same reason). So the bound is stated as two parts — 500 ppm of the pair's age, which is the kernel's own MAXFREQ on NTP's frequency discipline, and the step case that no arithmetic over the pair can see. That second part is deliberately not a method. Whether the robot's clock has been set is updater's judgement, and updater/src/preflight.rs already makes it, with a floor and the same reasoning. A copy here would be a second threshold to keep in step with the first, and it would sit in the one crate meant to stay nearly dependency-free because btd is part of the recovery path. The console's telemetry grows a clock row. It makes the same addition the browser cannot ask Rust for, and refuses to print an age that is obviously a missing sync rather than a slow network. It stays on the methods the page already uses rather than reaching for padStart or toTimeString, which exist nowhere in this repository. §11 said "wanted" and "not built", which stopped being true at v24 and was never corrected. It now says what was built, and what the check the section asked for found. abs-capture-time is a dead end — webrtcsink (0.15.3 and main), aiortc, and neither the WebRTC specification nor the stats one surface it — so frames are dated from RTCP sender reports, whose NTP time is the realtime clock by default. What the check turned up that the section did not expect is that this works for exactly one kind of receiver: dating a frame needs the report's RTP time as well as its NTP time, and only GStreamer exposes both (sr-rtptime alongside sr-ntptime). aiortc and every browser see the NTP half alone. Additive, so no API_VERSION bump. Co-authored-by: cyathea152 Assisted-by: DeepSeek v4.1 via DSH --- docs/design/remote-access-design.md | 2 +- docs/design/remote-webrtc.md | 92 +++++++++++++++-------- docs/design/robotd-design.md | 4 +- duck-ipc-proto/src/lib.rs | 109 ++++++++++++++++++++++++++++ mediad/src/session.rs | 13 ++-- mediad/webclient/index.html | 51 ++++++++++++- 6 files changed, 232 insertions(+), 39 deletions(-) diff --git a/docs/design/remote-access-design.md b/docs/design/remote-access-design.md index ab78dfed..f5a632de 100644 --- a/docs/design/remote-access-design.md +++ b/docs/design/remote-access-design.md @@ -1018,7 +1018,7 @@ Five slices, and the first two are independently useful and need no client: |---|---| | §2.4 the scope breadth | one public device-code client in the `pollen-robotics` HF org with `openid profile read-repos`, created by somebody with org admin. Not blocking — a scope change is a re-login — and it should not ship without it | | a calibration for the camera | `media.video` publishes the module's design figures with `calibrated: false`, which is enough to map a room and not enough for metrology. Measuring one robot and writing `[media.intrinsics]` closes it for that robot; a per-unit calibration in provisioning closes it for the family. §11 of `remote-webrtc.md` | -| everything on the wire should be timestamped at source | `remote-webrtc.md` §11: `abs-capture-time` on the media, checked against what `webrtcsink`, a browser and `aiortc` actually surface; and a monotonic-plus-epoch field on every control-channel notification that describes a moment. Wanted for any consumer that has to relate what the robot saw to what it felt — visual-inertial SLAM is the case that makes it concrete — and it wants its own version bump rather than riding along with a transport | +| everything on the wire should be timestamped at source | `remote-webrtc.md` §11. **The control half is built** (v24: `t_ns` on `robot.state`/`tof.frame`/`head_imu.frame`, and the `mono_ns`/`real_ns` pair `media.video` publishes to put them on the wall clock; no version bump was needed, being additive). **The media half was checked and the extension is a dead end**: `webrtcsink`, `aiortc`, and neither the WebRTC specification nor the stats one surface `abs-capture-time`. Frames are dated from RTCP sender reports instead, whose NTP time is the realtime clock by default — and that works only for a receiver that reads the report's **RTP** time as well as its NTP time, which is GStreamer (`sr-rtptime`). `aiortc` and every browser see the NTP half alone and cannot say *which* frame it was | | §2.6 `logout` revokes nothing | whether Hugging Face accepts a revocation for the first-party device-code client, checked rather than assumed. Not blocking — signing out stops the robot being reachable, and a stolen board is answered on hf.co — but it is the difference between "forgotten" and "revoked" | | §6 the relay check needs a token | `.github/workflows/turn-endpoint.yml` exists and runs daily, and fails until an `HF_TOKEN` secret is set on the repository — a Hugging Face token with no scope beyond sign-in, used only to mint TURN credentials. Failing loudly is deliberate; the alternative is a check that skips itself into silence | | §6 the relay is somebody else's Space | a credentials proxy of our own, holding the Cloudflare key in one place instead of trusting a dormant project's Space to keep its name. `--turn-url` is the seam. Not blocking — the Space answers — but `*.hf.space` is `{owner}-{space}` and there is no alias left to repoint if it moves | diff --git a/docs/design/remote-webrtc.md b/docs/design/remote-webrtc.md index d1b1b1f9..b8a290c5 100644 --- a/docs/design/remote-webrtc.md +++ b/docs/design/remote-webrtc.md @@ -539,35 +539,69 @@ the 1920×1080 crop, and every intrinsic would be off by about 1.7×. `mediad::c arithmetic and the mode table, including the fact that reading 720p off the sensor would *narrow* the view to 27° rather than saving anything. -## 11. Everything on the wire should carry the time it happened — **wanted** - -Nothing this transport carries is timestamped at source today. A frame arrives when it arrives, a -`robot.state` notification arrives when it arrives, and a consumer that wants to know *when* the -robot saw or felt something has only its own clock to go on — which, over a relay on another -continent, is off by whatever the path cost that second. - -That is fine for driving a robot you are watching, and it is the wrong shape for everything a -remote consumer is interesting for. **SLAM is the case that makes it concrete**: monocular SLAM on -a stream with no capture times can be run, and the moment somebody wants visual-inertial — the IMU -this robot already has, at 50 Hz, on the same control channel — the two series cannot be related -except by guessing. Timestamps applied at the far end measure the network, not the robot. - -Two halves, and they are not the same problem: - -- **Media.** RTP timestamps are relative to a random offset, so they order frames and date none of - them. The mechanism for this is the `abs-capture-time` RTP header extension, which carries a - wall-clock capture time per packet and is what a receiver needs to line video up against - anything else. Whether `webrtcsink` will negotiate it, and what a browser and `aiortc` expose of - it, is the thing to check first — a header extension nothing on the receiving side surfaces buys - nothing. -- **The control channel.** This one is ours and cheap: a monotonic reading, plus the boot epoch - that makes it comparable across processes, on every notification that describes a moment. The - cost is a field per message and an argument about which clock — and the answer has to be the - same one the media path ends up dating frames with, or the two series still cannot be joined. - -Not built, and deliberately not started as part of the remote path: it changes what every -notification looks like, so it wants its own decision and its own version bump rather than riding -along with a transport. `remote-access-design.md` §9 carries it as open. +## 11. Everything on the wire should carry the time it happened — **the control half built, the media half checked** + +Half of what this section wanted is built, and the other half has been checked rather than left +wanted. This is both, and what is left. + +**The control channel — built (v24).** `robot.state` and `tof.frame` carry `t_ns`, and so does +`head_imu.frame`: `CLOCK_MONOTONIC` in nanoseconds, one clock every daemon on a board shares, so a +sample from `robotd` and a frame from `tofd` go on one axis without an argument about two start +times. The **boot epoch** this section asked for is the second half of the pair `media.video` +publishes — `mono_ns` and `real_ns` — because the two clocks come off one hardware source, which +makes the mapping from any `t_ns` to wall clock an addition rather than an estimate. +`proto::clock::ClockPair` is the consumer's side of that pair, and it is where the bound this +section asked to have "written down somewhere" lives: the kernel's own ceiling on NTP's frequency +discipline, `MAXFREQ` at 500 ppm of the pair's age — 30 µs over a minute, 1.8 ms over an hour — +which covers slewing and **not** a step. That gap is not hypothetical here: the board has no +battery-backed RTC and boots at 1970, so a pair taken before NTP first sets the clock is wrong by +decades while looking like an ordinary number. Whether the clock has been set is `updater`'s +judgement rather than this bridge's — `updater/src/preflight.rs` owns that floor and reaches it the +same way — and a consumer that needs the answer should ask there rather than test this pair's +contents. `robotd-design.md` §Mapping telemetry owns the fields. + +**The media half — the check came back "not cheap".** The mechanism named here was the +`abs-capture-time` RTP header extension, and the first job this section set was finding out whether +anything on the receiving side surfaces it. Nothing does: + +- `webrtcsink` configures TWCC and color-space and nothing else, on 0.15.3 — the version + `media-bringup.md` builds — and on the newer main. It will not put a capture time on the wire. +- `aiortc` does not read one: its extension map carries the stream id, `abs-send-time`, the offset, + the audio level and the sequence number. +- Neither the WebRTC specification nor the stats one has a capture-time member: + `getSynchronizationSources()` has no capture timestamp, and `getStats()` — which is where such a + thing would live — has none either. + +So the extension buys nothing, and the answer is the one the control half already implies: **RTCP +sender reports**, whose NTP time is the realtime clock by default (`rtpmanager`'s `ntp-time-source` +is `ntp`), put on the monotonic axis by `media.video`'s pair. That is why v24 published +`mono_ns`/`real_ns` rather than waiting for an extension. + +**Only one kind of receiver can actually use it, and the difference is the RTP half of the pair.** +Dating a frame needs the sender report's *two* numbers — its NTP time and the RTP time it was taken +at: + +| Receiver | NTP from an SR | the RTP time it pairs with | +|---|---|---| +| GStreamer (`rtpbin` → `rtpsession`'s `stats`) | `sr-ntptime` | **`sr-rtptime`** | +| `aiortc` | `remoteTimestamp`, in stats | discarded | +| a browser | `remoteTimestamp`, in stats | not in any API | + +So a GStreamer receiver can map a decoded frame's RTP timestamp onto the wall clock and date it; an +`aiortc` peer and a browser page can say what the sender's clock read and cannot say which frame +that was. That is a limit of those two receivers rather than of the wire — the robot is already +sending the report — and it is why the console's clock row dates a *sample* and not a picture. + +**What is left.** A consumer that does it, in the one place it can be done: a GStreamer receiver +that reads `sr-rtptime`/`sr-ntptime` against a frame's RTP timestamp and reports which of that frame +and a `robot.state` sample came first. The console's clock row reads the control half of the bridge +today, so a sample's wall-clock moment and its age are on screen, bounded by the two machines' +clock agreement and not claimed to be exact. The error a frame carries is the sender-report interval +plus the encoder's latency, both of which such a receiver can state rather than inherit from the +network. `remote-access-design.md` §9 carries it. + +All of it is additive, so no `API_VERSION` bump was needed for the fields v24 added — an older +client ignores what it does not know. ## 12. Deferred, with reasons diff --git a/docs/design/robotd-design.md b/docs/design/robotd-design.md index 5bfe7489..75e179a6 100644 --- a/docs/design/robotd-design.md +++ b/docs/design/robotd-design.md @@ -1007,7 +1007,9 @@ projected gravity, and where the camera and the ToF sensor are. All three are ad `t` and `at_us` stay: they are each daemon's own elapsed time, and a reader that only has one stream still wants a number that starts at zero. `mediad`'s `media.video` answer reads `mono_ns` and `real_ns` at one instant, so RTP timestamps — which RTCP sender reports state in - wall-clock — can be put on the same axis. + wall-clock — can be put on the same axis. `proto::clock::ClockPair` is the consumer's side of + that pair: the two readings, and the addition that turns a `t_ns` into a wall-clock moment, + with its error bound rather than an estimate. - **`imu: {gyro, quat}`** is `ImuData` as the loop read it: the trunk IMU, 50 Hz, nothing above it (`docs/design/robotd-design.md` §IMU). The head IMU on the prototype HAT is not read by anything yet; when it is, it streams beside `tof.frame`, not here. diff --git a/duck-ipc-proto/src/lib.rs b/duck-ipc-proto/src/lib.rs index dbf68a94..84773ca3 100644 --- a/duck-ipc-proto/src/lib.rs +++ b/duck-ipc-proto/src/lib.rs @@ -5024,6 +5024,73 @@ pub mod clock { read(libc::CLOCK_REALTIME) } + /// The two clocks read together — the pair `media.video` carries as + /// `mono_ns` / `real_ns`. + /// + /// A consumer holding one of these pairs can put anything stamped with + /// [`super::RobotState::t_ns`] or [`super::TofFrame::t_ns`] on the wall-clock axis RTCP + /// sender reports live on. Both clocks come off the same hardware source, so the offset + /// between them is a constant while nothing is adjusting either one, and the mapping is an + /// addition. + /// + /// Two sides of one contract, and one method each: [`ClockPair::now`] is what a publisher + /// calls so the pair is genuinely read together, and [`ClockPair::wall`] is what a receiver + /// calls to use it. `wall` has no caller in this workspace — the receivers that need it are + /// the ones `remote-webrtc.md` §11 describes, a peer that reads RTP rather than a robot + /// daemon — which is the normal shape for a wire contract this crate owns on everyone's + /// behalf. + #[derive(Debug, Clone, Copy, PartialEq, Eq)] + pub struct ClockPair { + /// `CLOCK_MONOTONIC`, nanoseconds. + pub mono_ns: u64, + /// `CLOCK_REALTIME`, nanoseconds since the Unix epoch. + pub real_ns: u64, + } + + impl ClockPair { + /// Read both clocks, as close together as two syscalls get (tens of nanoseconds apart, + /// measured — not one instant, and not worth pretending to be one). + pub fn now() -> Self { + Self { + mono_ns: monotonic_ns(), + real_ns: realtime_ns(), + } + } + + /// The wall-clock time a `CLOCK_MONOTONIC` reading happened at. + /// + /// **The bound, and what it is not.** The offset this uses was true when the pair was + /// taken, and both clocks are frequency-disciplined by NTP, so the error grows with the + /// age of the pair at no more than the kernel's own limit on that discipline — `MAXFREQ` + /// is 500 ns/s (500 ppm, `include/linux/timex.h`), which is 30 µs over a minute and 1.8 ms + /// over an hour. That is the number to quote to a consumer asking how exact this is, and + /// it is a bounded worst case rather than an estimate: the clocks share a hardware source, + /// so ordinary operation sits far inside it. + /// + /// The bound covers **slewing**, and nothing covers a **step**: `CLOCK_REALTIME` "may + /// have discontinuities if the time is changed using `settimeofday(2)`" + /// (`clock_gettime(2)`), while `CLOCK_MONOTONIC` cannot go backwards. A pair taken before + /// one is simply wrong, by an amount no arithmetic over the pair can recover — and this + /// hardware does it, because the board has no battery-backed RTC and boots at 1970. + /// **Ask `updater`'s `preflight` whether the clock has been set** rather than testing + /// this pair's contents: it owns that judgement (`CLOCK_FLOOR_UNIX`), it reaches it the + /// same way, and a threshold repeated here would be the second copy of it. + /// + /// One more term this cannot see: comparing the result against a *third* machine's wall + /// clock adds that machine's own disagreement with the robot, which is a property of two + /// NTP clamps rather than of these two clocks, and is normally 1–50 ms — larger than + /// everything above. + /// + /// The offset is signed — a reading can predate the pair — so it is carried in `i128` + /// (exact for any two `u64` clocks), and the result is clamped rather than allowed to + /// wrap: a wall clock before the Unix epoch is not a time a peer can say anything about. + pub fn wall(&self, mono_ns: u64) -> u64 { + let offset = i128::from(mono_ns) - i128::from(self.mono_ns); + let wall = i128::from(self.real_ns) + offset; + wall.clamp(0, i128::from(u64::MAX)) as u64 + } + } + #[cfg(test)] mod tests { #[test] @@ -5033,6 +5100,48 @@ pub mod clock { assert!(b >= a); assert!(super::realtime_ns() > 1_600_000_000_000_000_000); } + + /// The one arithmetic a consumer relies on: the pair's own instant maps back to itself, + /// and the same offset carries both ways — a reading later than the pair, and a reading + /// from *before* it, which is what a sample queued behind the pair's read produces. + #[test] + fn a_pair_maps_a_reading_on_either_side_of_its_own_instant() { + let pair = super::ClockPair { + mono_ns: 10_000, + real_ns: 20_000, + }; + // The pair's own instant is the wall clock it was read with. + assert_eq!(pair.wall(10_000), 20_000); + // 1 µs later on the monotonic axis is 1 µs later on the wall clock. + assert_eq!(pair.wall(11_000), 21_000); + // Earlier readings map by the same offset, backwards. Without this a sample stamped + // before the pair collapsed onto the pair's instant and read as "just now". + assert_eq!(pair.wall(5_000), 15_000); + assert_eq!(pair.wall(0), 10_000); + } + + #[test] + fn wall_never_underflows_a_clock_near_the_wrap() { + let pair = super::ClockPair { + mono_ns: u64::MAX - 5, + real_ns: 0, + }; + // The offset is carried signed, so a reading past the pair's own instant still maps. + assert_eq!(pair.wall(u64::MAX), 5); + // And one from before the Unix epoch clamps rather than wrapping round. + assert_eq!(pair.wall(0), 0); + } + + /// Reading the pair's own instant back gives the wall clock it was read with, which is + /// the one identity a consumer relies on before trusting the offset for anything else. + #[test] + fn a_pair_maps_its_own_instant_back_to_the_clock_it_was_read_with() { + let pair = super::ClockPair::now(); + // Within the tens of nanoseconds the two `clock_gettime` calls are apart — measured + // on this machine, and the reason `now` does not claim to read one instant. + let delta = pair.wall(pair.mono_ns).abs_diff(pair.real_ns); + assert!(delta < 1_000_000, "round trip moved by {delta} ns"); + } } } diff --git a/mediad/src/session.rs b/mediad/src/session.rs index 6c65132e..c0137796 100644 --- a/mediad/src/session.rs +++ b/mediad/src/session.rs @@ -95,15 +95,18 @@ pub fn video_notification(video: &Video) -> String { /// One function for both, because a peer that asks and a peer that listens must be told the same /// thing — and the console does both, a push when the channel opens and a call when it is ready. fn video_params(video: &Video) -> serde_json::Value { + // The two clocks, read as one pair: RTCP sender reports state RTP time in wall-clock + // (`real_ns`), `robot.state`/`tof.frame` stamp with `mono_ns`'s clock. A peer that has both can + // put the picture on the robot's axis, and `ClockPair` is what says so — reading them here as + // two bare calls would put the claim that they belong together in this comment rather than in + // the type, and would leave the next caller free to read them a second apart. + let clock = proto::clock::ClockPair::now(); let mut params = serde_json::json!({ "width": video.width, "height": video.height, "rotate": video.rotate, - // The two clocks at one instant: RTCP sender reports state RTP time in wall-clock - // (`real_ns`), `robot.state`/`tof.frame` stamp with `mono_ns`'s clock. A peer that has both - // can put the picture on the robot's axis. - "mono_ns": proto::clock::monotonic_ns(), - "real_ns": proto::clock::realtime_ns(), + "mono_ns": clock.mono_ns, + "real_ns": clock.real_ns, }); // Absent rather than null when the geometry is unknown: a consumer reading a missing key knows // it must calibrate, where one reading `null` has to be told what that meant. diff --git a/mediad/webclient/index.html b/mediad/webclient/index.html index b9d43e21..fb06bbde 100644 --- a/mediad/webclient/index.html +++ b/mediad/webclient/index.html @@ -248,9 +248,13 @@

telemetry

health– battery– temperatures– + clock–

The frame is robot.subscribe at 2 Hz; the last three rows are - robot.health, polled, because none of it is on that stream.

+ robot.health, polled, because none of it is on that stream. The clock row puts + robot.state's t_ns on the wall clock, via the mono_ns/ + real_ns pair media.video publishes — the sample's moment and how + long ago it was true, bounded by the two machines' clock agreement rather than claimed.

@@ -1249,6 +1253,37 @@

telemetry

const limited = (move.limited_by || []).join(", "); $("t-applied").textContent = limited ? `${twist(move.applied)} — ${limited}` : twist(move.applied); $("t-applied").className = limited ? "warn" : ""; + + // The clock row consumes what `media.video` publishes. `t_ns` is CLOCK_MONOTONIC — the same + // axis `mono_ns` was read on — so `(t_ns - mono_ns)` added to `real_ns` is when this sample + // was true, in wall-clock, and `Date.now()` minus that is how long ago the robot saw it. + // Seconds-of-day rather than a formatted date: this page keeps to the methods it already uses, + // and the age is the number anything actually reads. + // + // The board has no battery-backed RTC and boots at 1970, so a pair taken before NTP first set + // the clock yields a 1970 time that looks like an ordinary number. The page does not carry its + // own floor for that — `updater`'s preflight already judges whether the robot's clock has been + // set, and a threshold repeated here would be a second copy to keep in step. What this does is + // refuse to print an age that is obviously a missing sync rather than a slow network. + const clockCell = $("t-clock"); + const IMPLAUSIBLE_AGE_MS = 86400000; // a day: nothing on this path is near it + if (videoClock && typeof state.t_ns === "number" && state.t_ns > 0) { + const wallMs = Math.round((videoClock.real_ns + (state.t_ns - videoClock.mono_ns)) / 1e6); + const age = Date.now() - wallMs; + if (Math.abs(age) > IMPLAUSIBLE_AGE_MS) { + // Said out loud rather than shown: "1970", or a fifty-year age, would read as a fault in + // this page rather than as a fact about the robot. + clockCell.textContent = "clock not synced"; + clockCell.className = "warn"; + } else { + const secondsOfDay = (wallMs % 86400000) / 1000; + clockCell.textContent = `t+${secondsOfDay.toFixed(3)}s · ${Math.round(age)} ms ago`; + clockCell.className = ""; + } + } else { + clockCell.textContent = "–"; + clockCell.className = ""; + } } // ── link quality ─────────────────────────────────────────────────────────────── @@ -1414,8 +1449,18 @@

telemetry

// over the video would be in; only the