Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Working in this repository

`CONTRIBUTING.md` is the reference: building, testing, layout, conventions, releasing. This page
is the short list of things that are easy to get backwards, and where the answer lives when they
are not here.

## Docs own mechanisms; one page each

`docs/README.md` assigns every mechanism to one design doc. When a fact belongs to a page listed
there, every other page says one sentence and links. When two pages disagree, the one that does
not own the mechanism is the bug — and when behaviour and a design doc disagree, the doc is the
bug. [`docs/faq.md`](docs/faq.md) is the task-shaped front door for someone building against a
robot rather than changing it.

## A consumer uses WebRTC. `media.stream` is the fallback

The robot publishes H.264 over WebRTC, and that is the default for anything consuming a duck's
camera — it is encrypted end to end, it carries the control channel on the same session, and it
has a return path. It works from a data centre because the robot offers a relay candidate
(`remote-access-design.md` §6).

`media.stream` — the robot dialling an outbound WebSocket and pushing frames to you — is the
fallback for a **program** consuming **frames only** on a **long-running** stream, where relay
metering is the thing that matters. It has no return path and no control channel.

This is worth stating because the repository reads the other way round if you only follow the
code: `media.stream` was built when the relay endpoint was dead and WebRTC genuinely could not
connect from a data centre, so its module doc argues its own case at length. That endpoint is
fixed. Do not conclude from the volume of prose that it is the preferred path.

## Never design around a version difference

One user, one robot. An old component's limits are a question to raise, not something to route
around — bump `API_VERSION` and name the install consequence. A version skew is logged and served,
never refused; only a genuinely missing route or an unknown parameter may refuse.

## Releases are how a fix reaches a robot

`main` being fixed is not a robot being fixed. Robots on the stable channel move when a release is
cut, and a dev build from a branch is superseded by the next `daemon-dev-main` the board's
six-hourly check finds. `docs/design/updater-design.md` owns the mechanism.
1 change: 1 addition & 0 deletions CLAUDE.md
4 changes: 4 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
The [README](../README.md) is the front door — what a microduck is, and where to go. If you have
one in front of you and want to drive it, start at the [cheat sheet](robot/cheatsheet.md).

[`faq.md`](faq.md) is the other front door: task-shaped questions from somebody building
*against* a duck rather than changing it — running a model too heavy for the board, getting the
camera into their own program, why a Space cannot connect.

It is also where a **publisher** starts: [`policy-manifest.md`](policy-manifest.md) is the
contract for a `manifest.json` beside a microduck `.onnx`, and it owns every field. The design
docs give the reasoning and point at it.
Expand Down
83 changes: 83 additions & 0 deletions docs/faq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# FAQ

Questions that come up when you want a duck to do something it does not do out of the box. The
design docs say how the machinery works; this says which piece to reach for.

## I want to run a model that is too heavy for the board

Run it somewhere else and send it the camera. The board is a Radxa Zero 3 — the NPU takes a small
detector and little more — so anything larger belongs off the robot, and a Hugging Face Space on
paid hardware is the path with the least to build: it already has a GPU option, an account system
the robot shares, and a URL.

**Use WebRTC.** The robot publishes H.264 over a WebRTC session, your Space consumes it, and the
rendezvous introduces the two so neither needs to know the other's address. `spaces/vision-demo/`
is the worked example.

```
your Space ──sign in, list robots──► rendezvous ◄──registers── the duck
your Space ◄═══════════ H.264 over WebRTC, media and control ════════════► the duck
```

Three things you get for free by staying on WebRTC, and they are the reason it is the default:
the media is encrypted end to end by DTLS-SRTP, the control channel rides the same session so you
can *drive* as well as watch, and there is a return path — audio, or a second video track — the
day you want one.

## Does it work from a data centre? My Space is behind a NAT I do not control

Yes. A robot behind a home router and a container behind a data centre's NAT usually cannot
hole-punch to each other, so the robot offers a **relay candidate** — short-lived Cloudflare
credentials it mints with its own account token — and your consumer uses it without holding any
credentials of its own. `remote-access-design.md` §6 is the mechanism.

Two caveats worth knowing before you are surprised by them:

- **A relay costs the robot owner's bandwidth**, metered per Hugging Face account at 10 GB a
month on the free tier. A continuous 720p stream is roughly a gigabyte an hour, so a Space that
watches all day will spend it. ICE only relays when it must — a direct pair is used whenever
one can be found — but "must" is common between a home and a data centre.
- **A browser consumer needs relay credentials of its own** when it has no IPv4 of its own; a
Python one does not. §6 has the case.

## My consumer is a program and the stream runs all day. Is there something cheaper?

There is a fallback, and it is a fallback rather than a second design: **`media.stream`**, where
the robot dials *your* WebSocket and pushes frames outbound.

```
your Space ──media.stream {url: "wss://…/frames"}──► rendezvous ──► the duck
the duck ══════════ frames, outbound wss, direct ══════════════► your Space
```

It costs nobody's relay, because an outbound connection from the robot needs no hole punched. Use
it when all three are true: the consumer is a **program** and not a person, the stream is
**long-running** enough for relay metering to matter, and you need **frames only**.

What you give up by leaving WebRTC, which is why it is not the default:

- **No return path at all.** Nothing reaches the robot on this transport — no audio, no second
track, no teleop loop.
- **No control channel.** Driving the robot means a JSON-RPC call over the rendezvous
(`spaces/shared/wire.py`), separately.
- **Encryption is your TLS, not DTLS-SRTP**, and it terminates at your server rather than at the
peer.

The robot sends one text **hello** describing what is coming — `frames.encoding` is `h264` or
`jpeg` — then one binary message per frame. Branch on the hello rather than sniffing the bytes;
`spaces/vision-demo/receiver.py` is the reference receiver and `decoder_for` is the branch.

## How does my Space find the robot, and what stops somebody else's reaching it?

The account. The robot registers with the rendezvous holding its own Hugging Face token, your
Space signs the visitor in with `hf_oauth`, and the service only ever shows an account the robots
it owns. Nothing is configured on the robot and no robot-side gate is involved —
`remote-access-design.md` §7 is the argument for why that is enough.

`spaces/shared/rendezvous.py` is the listing call: `ducks(token)` gives you what that account can
reach.

## Can I test without a robot?

`spaces/vision-demo/fake_duck.py` stands in for one, and `scripts/duck-sim` runs the real daemons
against a MuJoCo body ([`robot/simulation.md`](robot/simulation.md)).
40 changes: 21 additions & 19 deletions mediad/src/stream.rs
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
//! Frames out to a WebSocket **this robot dials**, for a Space that runs a model on them.
//!
//! # Why the robot dials, and why that is the whole idea
//! # This is the fallback, and WebRTC is the default
//!
//! The goal is a Space on Hugging Face hardware processing this camera. The obvious route is the
//! one `vision-demo` takes — a WebRTC consumer pulls the stream through the rendezvous — and it
//! runs into the one thing WebRTC cannot do without help: a robot behind a home router and a
//! container behind a data centre's NAT need a **relay candidate** to pair
//! (`remote-access-design.md` §6). Signalling crosses; media needs somebody's relay.
//! **A consumer should use WebRTC**, which carries encrypted media, a control channel on the same
//! session and a return path, and which reaches a data centre because the robot offers a relay
//! candidate (`remote-access-design.md` §6). `docs/faq.md` is the decision, in the shape somebody
//! arrives at it. This module is for the narrow case WebRTC serves badly: a **program** consuming
//! **frames only** on a **long-running** stream, where a relay's metered bandwidth is the cost
//! that matters.
//!
//! An outbound WebSocket needs nobody's. **The robot already proves this every second it is
//! reachable**: `relay.rs` holds an outbound HTTPS stream to a Space right now, and nothing about
//! a home router objects. So the frames go the same way the registration does — outward — and NAT
//! stops being a participant.
//! In that case an outbound WebSocket needs nobody's relay. **The robot already proves this every
//! second it is reachable**: `relay.rs` holds an outbound HTTPS stream to a Space right now, and
//! nothing about a home router objects. So the frames go the same way the registration does —
//! outward — and NAT stops being a participant.
//!
//! ```text
//! Space ──media.stream {url: "wss://…/frames"}──► rendezvous ──► this robot
Expand All @@ -22,18 +23,19 @@
//! this scale where relaying payload through a shared service would not: one small envelope per
//! session, on a service the mini fleet also depends on, and the bytes go point to point.
//!
//! **This is not a workaround for a relay that does not exist**, and it used to read like one.
//! `turn.rs` offers relay candidates and they work. The argument above is the argument either
//! way: a relay is metered per Hugging Face account, so every one of these frames would be spent
//! against an allowance (§6: 10 GB a month) that the robot's owner also needs for being
//! *watched* — and the shortest path between a board and a data centre is not through a third
//! one. A relay is the fallback for a session that cannot be made direct. This one can.
//! **This was written when the relay endpoint was dead and WebRTC could not connect from a data
//! centre at all.** That is fixed (§6), so the reason this exists is now the narrow one above and
//! not "the alternative does not work". What survives of the original argument is the cost: a
//! relay is metered per Hugging Face account at 10 GB a month, and a stream that runs all day
//! spends an allowance its owner also needs for being *watched*.
//!
//! # What it is not
//!
//! Not a replacement for a relay candidate in general. There is no return media path, so nothing
//! here helps a browser *watch* a robot, carries audio, or closes a teleop loop — a viewer wants
//! WebRTC and §6 is still what it needs. This is for the case where the consumer is a program.
//! **Not a replacement for WebRTC, and not the path to reach for first.** There is no return
//! media path and no control channel, so nothing here helps a browser *watch* a robot, carries
//! audio, or closes a teleop loop; driving means a separate JSON-RPC call over the rendezvous.
//! Encryption is the receiver's TLS rather than DTLS-SRTP, terminating at a server instead of at
//! the peer. A consumer that is not all three of program, frames-only and long-running wants §6.
//!
//! # This half is portable, and that is deliberate
//!
Expand Down
Loading