mediad(turn): the relay endpoint is the Space, not the alias in front of it

.github/workflows/turn-endpoint.yml

@@ -0,0 +1,77 @@
+# Does the relay credentials endpoint still answer?
+#
+# `mediad::turn` fetches short-lived Cloudflare credentials so a robot can offer a `relay`
+# candidate to a consumer that cannot reach it directly. The endpoint it used before
+# 2026-09 had been dead since June — a dangling Route53 delegation — and **nothing
+# noticed for three months**, because the only symptom is a warning in a journal nobody
+# reads and a candidate type nobody counts. Every other check in this repository passes
+# regardless: they pair two peers on one network, which never looks at a relay.
+#
+# So this is the one check that would have caught it, and it is deliberately *not* on
+# `pull_request`. The failure being guarded against is "nobody touched this for months",
+# which a PR trigger cannot see; and a third party's outage must never block work that has
+# nothing to do with it. A scheduled failure mails whoever owns the repository, which is
+# the right blast radius for "an upstream service went away".
+name: turn-endpoint
+
+on:
+  schedule:
+    # Daily. The endpoint is somebody else's Space, and a week of relay coverage lost
+    # before anybody hears about it is most of the damage already done.
+    - cron: "41 5 * * *"
+  workflow_dispatch:
+
+jobs:
+  credentials:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      # **Read the URL out of the source rather than repeating it here.** A check with its
+      # own copy of the endpoint tests whatever it was last told, which can drift from what
+      # the daemon compiles in — and a green check on an endpoint no robot uses is worse
+      # than no check, because it reads as proof.
+      - name: Ask the endpoint the daemon uses
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+        run: |
+          set -euo pipefail
+
+          url=$(sed -n 's/^pub const DEFAULT_TURN_ENDPOINT: &str = "\(.*\)";$/\1/p' \
+              mediad/src/turn.rs)
+          [ -n "$url" ] || {
+              echo "::error::could not read DEFAULT_TURN_ENDPOINT from mediad/src/turn.rs"
+              exit 1
+          }
+          echo "endpoint: $url"
+
+          # No token means this cannot check anything. Failing is the point: a check that
+          # skips itself into permanent silence is the failure mode this workflow exists
+          # to end.
+          [ -n "${HF_TOKEN:-}" ] || {
+              echo "::error::no HF_TOKEN secret. Add a Hugging Face token with no scopes" \
+                   "beyond sign-in; it is used only to mint TURN credentials."
+              exit 1
+          }
+
+          code=$(curl -sS -o body.json -w '%{http_code}' --max-time 30 \
+              -H "Authorization: Bearer $HF_TOKEN" "$url?ttl=600")
+          [ "$code" = "200" ] || {
+              echo "::error::$url answered HTTP $code, not 200"
+              head -c 400 body.json || true
+              exit 1
+          }
+
+          # A 200 carrying no relay is still a robot with no relay candidate to offer —
+          # which is the outage, not a milder version of it. `stun:` entries do not count:
+          # `webrtcbin` takes a STUN server through its own property, and this is about the
+          # thing STUN cannot do.
+          relays=$(jq '[.iceServers[]? | .urls] | flatten
+                       | map(select(startswith("turn:") or startswith("turns:")))
+                       | length' body.json)
+          echo "relay servers offered: $relays"
+          [ "$relays" -gt 0 ] || {
+              echo "::error::$url answered 200 with no turn:/turns: entries"
+              head -c 400 body.json || true
+              exit 1
+          }

Cargo.toml

@@ -90,6 +90,7 @@ tracing = "0.1"
 tracing-subscriber = "0.3"
 tokio-util = "0.7"
 async-trait = "0.1"
+url = "2"
 
 # No custom [profile.release]. Binary size is not worth optimising for: model artifacts
 # will dwarf a few MB of binary, and the tuning cost more than it bought —

docs/design/remote-access-design.md

@@ -762,10 +762,11 @@ error, deliberately — safety refusing to run a policy on a fallen robot is not
 page that only catches errors reports every one of those as a success and leaves a motionless
 robot unexplained.
 
-**And the transport is the one thing this cannot prove from a Space yet.** The control channel is
-SCTP over the same candidate pair as the media, so §6's dead TURN endpoint takes the click with
-it: from a data centre the session negotiates and may then carry nothing. That is why the status
-line names the stage it reached rather than saying "connecting…" — and why the page has a second
+**And the transport is what this leans on hardest from a Space.** The control channel is SCTP over
+the same candidate pair as the media, so a relay that is not there takes the click with it: from a
+data centre the session negotiates and may then carry nothing. §6's endpoint answers now, so the
+ordinary case is covered — but a metered dependency is still a dependency, which is why the status
+line names the stage it reached rather than saying "connecting…", and why the page has a second
 way in.
 
 **`lan.py` is that second way, and it is a transport rather than a second design.** The robot is
@@ -856,11 +857,11 @@ matters more than it sounds, because `aiortc`'s STUN client works where its TURN
 so a Python consumer *cannot* be the side that relays. `reachy_mini`'s #1182 established this
 arrangement and `mediad::turn` is the same one.
 
-The credentials are Cloudflare's, minted per account by a proxy Hugging Face hosts
-(`turn.fastrtc.org/credentials`) and authenticated with **the same token the relay signs in with**
-— so a robot that belongs to somebody can offer a relay and one that belongs to nobody cannot,
-which is the same line §2 draws everywhere else. They are short-lived: a task refreshes at half of
-a 600 s lifetime, and retries in thirty seconds after a *transient* failure only. A robot nobody
+The credentials are Cloudflare's, minted per account by a proxy Hugging Face hosts and
+authenticated with **the same token the relay signs in with** — so a robot that belongs to
+somebody can offer a relay and one that belongs to nobody cannot, which is the same line §2 draws
+everywhere else. They are short-lived: a task refreshes at half of a 600 s lifetime, and retries
+in thirty seconds after a *transient* failure only. A robot nobody
 has signed in has nothing to retry for, and a warning every thirty seconds for the life of the
 daemon is how a log stops being read.
 
@@ -872,22 +873,61 @@ blocks (a `try_read` that yields nothing rather than waiting) and never fails. A
 the ordinary state for the first few seconds after boot and forever on a robot with no account,
 and it means host and srflx only, which is all anything on the same network needs.
 
-**And the endpoint is not answering, which is where this stands.** `turn.fastrtc.org` has no A
-record and `fastrtc.org` has no NS records at all, from three public resolvers and from the board
-— so the proxy both `fastrtc`'s own current code and `reachy_mini`'s #1182 point at cannot be
-reached by anybody. Their documentation describes it as a live Hugging Face–Cloudflare arrangement
-(10 GB a month free with an account), so this reads as a lapsed registration or an outage rather
-than a moved URL, and it means the mini fleet's relay path is down too. Worth telling whoever owns
-`fastrtc`.
-
-Three ways on, in the order they should be considered:
-
-- **Wait, having reported it.** The robot degrades exactly as designed — a warning every thirty
-  seconds and host/srflx candidates — so nothing is broken except reaching a robot from a network
-  that needs a relay.
-- **Our own proxy**, which is what that endpoint is: a small service holding a Cloudflare Calls key
-  and minting short-lived credentials for a caller presenting a valid HF token. `--turn-url` is
-  already the seam it plugs into, and the key stays in one place rather than on robots.
+**The proxy is the Space, and the name in front of it was the dead part.** `turn.fastrtc.org` —
+what `fastrtc`'s own code points at and what `reachy_mini` #1182 copied into this arrangement — is
+a dangling delegation, not an outage: the `.org` registry names four Route53 nameservers for the
+zone, the registration is healthy and locked until 2027, and all four nameservers answer `REFUSED`
+for the zone they are authoritative for, which is what Route53 says when the hosted zone behind
+them is gone. `fastrtc/turn-service`, the Space that alias pointed at, never stopped answering.
+So `DEFAULT_TURN_ENDPOINT` addresses it directly, at
+`https://fastrtc-turn-service.hf.space/credentials`, and the vanity record is out of the path.
+
+**That also closes a token-exfiltration route, which is the half that mattered more than the
+outage.** A dangling Route53 delegation is a known takeover shape — create hosted zones until AWS
+assigns you one of the four delegated nameservers, and one is enough, because a resolver needs
+only one authoritative answer. Whoever landed it would serve records for the name, pass DNS
+validation for a certificate on it, and be handed the account token every signed-in robot sends
+as a bearer header every five minutes. That token is the robot's whole credential; §2.4 is about
+how broad its scopes are.
+
+**The endpoint is checked before the token can reach it.** `turn::parse_endpoint` is a `clap`
+`value_parser` on `--turn-url`: `https` unless the host is loopback, no userinfo, no query and no
+fragment. A wrong value stops the daemon at argument parsing rather than becoming a warning every
+thirty seconds. Redirects need no separate guard — `reqwest` strips `Authorization` when a
+redirect crosses scheme, host or port (`src/redirect.rs`, `remove_sensitive_headers`), so the
+token cannot be walked to a third-party origin.
+
+**The allowance is the robot owner's, and it is finite** — 10 GB a month on a free Hugging Face
+account. A relayed video session is roughly a gigabyte an hour, so a robot driven hard over a
+relay can spend it, after which the proxy answers with no relay servers and `turn.rs` logs "the
+TURN proxy offered no relay servers" at info level. That is indistinguishable from a robot that
+was never offered one, which is a legibility gap worth closing when somebody hits it. It is also
+the argument `stream.rs` makes for sending frames outbound rather than through a relay.
+
+**Something now notices when it dies, which is the reason this went unnoticed for three months.**
+The endpoint was already dead when #1182 shipped it, and the only symptom was a warning in a log
+and a candidate type nobody counted. `.github/workflows/turn-endpoint.yml` runs daily: one
+authenticated `GET` against `DEFAULT_TURN_ENDPOINT`, asserting 200 and at least one `turn:`/
+`turns:` entry. Every other check in the repository passes regardless, because every one of them
+pairs two peers on one network, which never looks at a relay.
+
+Three things about its shape are deliberate. It reads the URL **out of `turn.rs` with `sed`**
+rather than keeping a copy, because a check holding its own endpoint tests whatever it was last
+told and can drift from what the daemon compiles in — and a green check on a URL no robot uses is
+worse than no check, since it reads as proof. It is **not** on `pull_request`: the failure being
+guarded against is "nobody touched this for months", which a PR trigger cannot see, and a third
+party's outage must never block unrelated work. And a **missing `HF_TOKEN` secret fails** rather
+than skipping, because a check that quietly skips itself into permanent silence is the exact
+failure mode it exists to end.
+
+Two things this deliberately does not do:
+
+- **Our own proxy.** This endpoint is a small service holding a Cloudflare Calls key and minting
+  short-lived credentials for a caller with a valid HF token, and running one ourselves would end
+  the dependency on a dormant project's Space — `--turn-url` is already the seam it plugs into.
+  Worth doing, and not worth blocking relay coverage on: `*.hf.space` is `{owner}-{space}`, so it
+  breaks if `fastrtc` renames or removes the Space, and there is no CNAME layer left to repoint.
+  That is the residual risk, named rather than closed.
 - **A Cloudflare key on the robot**, using `TURN_KEY_ID` and `TURN_KEY_API_TOKEN` directly. Fastest
   and worst: a long-lived API token on every board, which is the shape of mistake §2.4 exists to
   stop making.
@@ -949,8 +989,10 @@ Five slices, and the first two are independently useful and need no client:
    bridge to, which is a robot whose pipeline never reached PLAYING.
 5. **STUN decided; TURN offered by the robot.** §6. **Done**: `stun.l.google.com:19302` on both
    ends, and `mediad::turn` keeps Cloudflare credentials fresh so every consumer's offer carries a
-   `relay` candidate. §6 has the argument; the two things to carry away are that only the robot
-   needs credentials, and that reading them must never block the thread building an offer.
+   `relay` candidate — from `fastrtc-turn-service.hf.space`, the Space itself, because the alias
+   in front of it is a dangling delegation. §6 has the argument; the three things to carry away
+   are that only the robot needs credentials, that reading them must never block the thread
+   building an offer, and that no check in this repo would notice if that endpoint died again.
 
 ## 9. What is open, and who can close it
 
@@ -960,12 +1002,15 @@ Five slices, and the first two are independently useful and need no client:
 | a calibration for the camera | `media.video` publishes the module's design figures with `calibrated: false`, which is enough to map a room and not enough for metrology. Measuring one robot and writing `[media.intrinsics]` closes it for that robot; a per-unit calibration in provisioning closes it for the family. §11 of `remote-webrtc.md` |
 | everything on the wire should be timestamped at source | `remote-webrtc.md` §11: `abs-capture-time` on the media, checked against what `webrtcsink`, a browser and `aiortc` actually surface; and a monotonic-plus-epoch field on every control-channel notification that describes a moment. Wanted for any consumer that has to relate what the robot saw to what it felt — visual-inertial SLAM is the case that makes it concrete — and it wants its own version bump rather than riding along with a transport |
 | §2.6 `logout` revokes nothing | whether Hugging Face accepts a revocation for the first-party device-code client, checked rather than assumed. Not blocking — signing out stops the robot being reachable, and a stolen board is answered on hf.co — but it is the difference between "forgotten" and "revoked" |
+| §6 the relay check needs a token | `.github/workflows/turn-endpoint.yml` exists and runs daily, and fails until an `HF_TOKEN` secret is set on the repository — a Hugging Face token with no scope beyond sign-in, used only to mint TURN credentials. Failing loudly is deliberate; the alternative is a check that skips itself into silence |
+| §6 the relay is somebody else's Space | a credentials proxy of our own, holding the Cloudflare key in one place instead of trusting a dormant project's Space to keep its name. `--turn-url` is the seam. Not blocking — the Space answers — but `*.hf.space` is `{owner}-{space}` and there is no alias left to repoint if it moves |
 
 Closed since this page was written: the OAuth client (§2.3 — Hugging Face ships one), whether the
 token expires (§2.7 — thirty days, with a rotating refresh token), which rendezvous to use (§4 —
 the mini's), whether we can read it (§4 — we maintain it; the "private repo" in an earlier draft
-was a wrong-name 401), and where the client is served (§5 — a static Space with `hf_oauth`, because
-the question was never hosting but how a page gets a token).
+was a wrong-name 401), where the client is served (§5 — a static Space with `hf_oauth`, because
+the question was never hosting but how a page gets a token), and which relay endpoint to use (§6 —
+the Space itself; the alias in front of it is a dangling delegation).
 
 One item this page created and closed: **peers are keyed by token**, so two things sharing one
 take turns being reachable. Not a provisioning problem — images are built from scratch, not cloned,

mediad/Cargo.toml

@@ -50,6 +50,11 @@ reqwest = { version = "0.13.4", default-features = false, features = ["rustls",
 # reconnection field. Sixty lines to hand-roll and one of them would be the bug: a `data:` split
 # across two TCP reads.
 eventsource-stream = "0.2.3"
+# Parsing, not fetching: the one endpoint this daemon sends the account token to is
+# checked before the token can reach it, and "is this loopback" and "does this carry
+# userinfo" are the two questions a hand-rolled split on `:` gets wrong. Already in the
+# lock via `reqwest`.
+url.workspace = true
 # `StreamExt`, for the byte stream `reqwest` hands over. Already in the lock via `updater`.
 futures-util = "0.3.33"
 # The local half of the bridge: a WebSocket client to `webrtcsink`'s own signalling server on

mediad/src/main.rs

@@ -55,7 +55,11 @@ struct Args {
     /// Hugging Face hosts this proxy and mints Cloudflare credentials for the account the token
     /// belongs to, which is why offering a relay needs no new secret on the robot. A flag for
     /// pointing a board at a fake; there is nothing to choose on a real one.
-    #[arg(long, default_value = mediad::turn::DEFAULT_TURN_ENDPOINT)]
+    ///
+    /// Checked here rather than trusted: this is the one URL the account token is sent to, and
+    /// `parse_endpoint` says what it will not send it over.
+    #[arg(long, default_value = mediad::turn::DEFAULT_TURN_ENDPOINT,
+          value_parser = mediad::turn::parse_endpoint)]
     turn_url: String,
 
     /// Do not register with the rendezvous service, whatever the token file says.

mediad/src/relay.rs

@@ -21,10 +21,10 @@
 //!
 //! The bridge above carries a *negotiation*: SDP and ICE, so that a consumer and the robot can
 //! find a path between them and speak WebRTC over it. When they cannot find one, everything built
-//! on it is gone — and right now they frequently cannot, because a relay candidate needs
-//! `turn.fastrtc.org`, which has no A record and whose zone has no NS records (§6). Signalling
-//! crosses, media does not, and the control channel goes with it because SCTP rides the same
-//! candidate pair.
+//! on it is gone. A relay candidate is what keeps that from happening (§6) — and it is a
+//! dependency rather than a guarantee: it is somebody else's service, metered per account, and
+//! the control channel is SCTP over whatever pair ICE settled on, so a relay that stops being
+//! available takes a JSON-RPC call of a few hundred bytes down with the video.
 //!
 //! So the JSON-RPC a consumer wants to send does not have to go through WebRTC at all, and the
 //! rendezvous turns out to already carry it: `handle_peer_message` in their `app.py` relays