mediad(turn): the relay endpoint is the Space, not the alias in front of it - #258
Merged
Merged
Conversation
… of it `DEFAULT_TURN_ENDPOINT` pointed at `turn.fastrtc.org`, which `fastrtc`'s own code points at and `reachy_mini` #1182 copied. That name is a dangling delegation rather than an outage: the `.org` registry names four Route53 nameservers for the zone, the registration is healthy and locked until 2027, and all four nameservers answer REFUSED for the zone they are authoritative for — what Route53 says when the hosted zone behind them is gone. The Space that alias pointed at, `fastrtc/turn-service`, never stopped answering. So address it directly. Every microduck robot has been offering host and srflx candidates only, which is why a consumer behind symmetric NAT or CGNAT could not reach one at all, and why demos passed regardless: two peers on one network pair on host candidates and never look at a relay. It also takes a name that can be taken over out of the path a signed-in robot sends its account token down. A dangling Route53 delegation is a known takeover shape, and whoever landed one of those four nameservers would serve records for the name, pass DNS validation for a certificate on it, and be handed the token every five minutes. `turn::parse_endpoint` checks `--turn-url` before the token can reach it — https unless the host is loopback, no userinfo, no query, no fragment — as a clap `value_parser`, so a wrong value stops the daemon at argument parsing instead of becoming a warning every thirty seconds. Redirects need no guard: `reqwest` strips `Authorization` when one crosses scheme, host or port. Nine places in the prose reasoned from "there is no relay candidate", and two of them used it as the justification for a design. `stream.rs` still sends frames outbound and `lan.py` is still a LAN transport, but for the reason that was always the real one — a relay is metered against the robot owner's allowance, and the shortest path between a board and a data centre is not through a third one — rather than because the relay was down. §6 of remote-access-design.md records what this does not close: a relay-only connectivity check, without which nothing here would notice the endpoint dying again, and a credentials proxy of our own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Coverage71.62% lines on this branch, against a floor of 70%. Per-file |
The endpoint `mediad::turn` fetched from was dead for three months before anybody noticed, and it was already dead when the default that named it shipped. Nothing in this repository would have caught that: the only symptom is a warning in a journal and a missing candidate type, and every check we have pairs two peers on one network, which never looks at a relay. One authenticated GET, asserting 200 and at least one turn:/turns: entry. Three deliberate choices. The URL is read out of turn.rs with sed rather than copied, because a check carrying its own endpoint drifts from what the daemon compiles in, and a green check on a URL no robot uses reads as proof while proving nothing. It is scheduled rather than on pull_request: the failure being guarded against is nobody touching this for months, which a PR trigger cannot see, and somebody else's outage must not block unrelated work. And a missing HF_TOKEN fails rather than skips, because a check that skips itself into permanent silence is precisely the failure mode it exists to end. It will fail until an HF_TOKEN secret is set on the repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
mediad/src/turn.rs'sDEFAULT_TURN_ENDPOINTishttps://turn.fastrtc.org/credentials, andmediad.servicepasses no--turn-url, so that is what every microduck robot uses. It has notresolved since at least 2026-06-04 (gradio-app/fastrtc#429,
still unanswered).
It is a dangling delegation, not an outage. The
.orgregistry names four Route53 nameserversfor
fastrtc.org, the registration is healthy and locked until 2027, and all four answerREFUSEDfor the zone they are authoritative for — which is what Route53 says when the hostedzone behind them is gone.
Two costs. Every signed-in robot offers host and srflx candidates only, so a consumer behind
symmetric NAT, CGNAT or a UDP-blocking firewall cannot reach it at all and there is no fallback,
because the fallback is the thing that is down. And a dangling Route53 delegation is a known
takeover shape: whoever wins a race to be assigned one of those four nameservers serves records
for the name, passes DNS validation for a certificate on it, and is handed the account token every
robot sends as a bearer header every five minutes.
Demos pass regardless, which is why this went unnoticed through
reachy_mini#1182 and since:two peers on one network pair on host candidates and never look at a relay.
Repro
SERVFAIL, thenREFUSEDfrom a nameserver the registry names as authoritative.Solution
The service never stopped answering — only the name in front of it did.
fastrtc/turn-service,the Space that alias pointed at, is up and gated on a bearer token:
401. So address the Space directly and take the vanity record out of the path.reachy_mini#1408 makes the same move and measured
a relay pair carrying ~2.2 Mbps of video through
turn.cloudflare.com.turn::parse_endpointchecks the endpoint before the token can reach it —httpsunless thehost is loopback, no userinfo, no query, no fragment — as a clap
value_parser, so a wrong valuestops the daemon while somebody is still looking at the terminal rather than becoming a warning
every thirty seconds in a log nobody reads. Redirects need no separate guard:
reqweststripsAuthorizationwhen one crosses scheme, host or port (src/redirect.rs,remove_sensitive_headers), so the token cannot be walked to a third-party origin.urlis a newdirect dependency of
mediadand was already in the lock viareqwest— one line ofCargo.lock.Nine places in the prose reasoned from "there is no relay candidate", and two used it as the
justification for a design.
stream.rsstill sends frames outbound andlan.pyis still a LANtransport — but for the reason that was always the real one, which is that a relay is metered
against the robot owner's Hugging Face allowance and the shortest path between a board and a data
centre is not through a third one. Not because the relay was down.
Verification
cargo test -p mediad— 79 pass, including three new ones on the guard: the default is a value itaccepts and survives the round trip unchanged (a default its own check rejects would stop every
robot at once), six destinations that would leak the token are refused, and loopback
httpstillworks because every other test in the module dials one.
cargo clippy -p mediad --all-targetsandcargo fmt --checkclean,cargo check --workspaceclean.End to end through clap:
Verified on hardware (olducky, 2026-09-14). The drop-in that had been overriding
--turn-urlwas removed first, so both binaries ran on their own default, same board, same network, same
account, eleven seconds apart:
PID 2483 is the released build falling back to the dead default, which is what every robot in the
fleet does today. PID 3033 is this branch.
systemctl show mediad -p ExecStartcarries no--turn-url, so the credentials came from the default — the one line this changes.And it reaches the wire:
mediad::pipeline: offering relay candidates relays=5on every sessionof a subsequent remote test, and a phone on 4G now connects through the console (that last step
also needed #280 — see there for why the robot's relay alone was not enough).
What this does not close, recorded in §6 and §9
A relay-only connectivity check— added,.github/workflows/turn-endpoint.yml. Daily,one authenticated
GETagainstDEFAULT_TURN_ENDPOINT, asserting 200 and at least oneturn:/turns:entry. It reads the URL out ofturn.rswithsedrather than keeping a copy,so it can never go green against an endpoint no robot uses. Scheduled and not on
pull_request, because the failure guarded against is "nobody touched this for months" and athird party's outage must not block unrelated work. It fails until an
HF_TOKENsecret isset on the repository — deliberately, since a check that skips itself into silence is the
failure mode it exists to end.
*.hf.spaceis{owner}-{space}, so this breaks iffastrtcrenames or removes the Space, and there is no alias layer left to repoint. Named asresidual risk rather than closed.
with no relay servers, and
turn.rslogs that at info level — indistinguishable from a robotthat was never offered one.
Branch
turn-endpoint-is-dead(unmerged, no PR) recorded the DNS findings and recommendeddropping the default so a relay became opt-in. This supersedes it: opt-in coverage is zero
coverage, and the endpoint that would provide it turns out to answer. That branch can be deleted.
🤖 Generated with Claude Code