Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,6 @@ secret*

# Playground and scratch state (see updater/examples/playground.rs).
/verify

# Owner-specific notes and locally captured robot data never belong in this public repository.
/.private/
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ one in front of you and want to drive it, start at the [cheat sheet](robot/cheat
| [`pair-a-gamepad.md`](robot/pair-a-gamepad.md) | Once per pad: pairing mode, `pad pair`, and what to do when it will not bond. |
| [`cheatsheet-dev.md`](robot/cheatsheet-dev.md) | The commands that need a dev board: branch builds, candidates, dev pushes. |
| [`dev-push.md`](robot/dev-push.md) | Build on your machine and install on the board over ssh, with no CI run. |
| [`model-channel.md`](robot/model-channel.md) | Package, verify and trial a policy bundle. |
| [`duckctl.md`](robot/duckctl.md) | Every `duckctl` command — the robot from a laptop, over Bluetooth. |
| [`install-dev.md`](robot/install-dev.md) | Setting up a board for development, from nothing. |
| [`install-by-hand.md`](robot/install-by-hand.md) | The same install as separate commands, for testing one step at a time. |
Expand Down
21 changes: 12 additions & 9 deletions docs/project/npu-bringup.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,15 +99,18 @@ that as the price of perception, the two should be measured apart.

## What is still missing

**Nothing on the robot can get a frame.** `mediad` has a raw NV12 tee branch that exists precisely
for this — `architecture.md` §5.3 — but no IPC exposes it, which is also why capturing a dataset has
to stop `mediad` to take the camera. Two ways forward, and they are not exclusive:

- **`media.frame`**: a call that answers with one frame. Useful for far more than perception (a
snapshot in the console, a still for a bug report), and it makes capture stop fighting the daemon.
- **The detector inside `mediad`**: subscribe to the raw branch, run the model at a few Hz, and
publish detections on the state stream. This is where it ends up — perception next to the sensor,
deriving features rather than shipping pixels — and it is what a behaviour would consume.
`media.frame` now exposes the raw **UYVY** tee branch locally at `/run/mediad/media.sock`.
`robotctl media frame --output frame.uyvy` asks for the latest frame without stopping `mediad` or
back-pressuring its encoder. The response starts with a JSON-RPC header (geometry, format, byte
count and capture time), followed by exactly that many binary bytes. It is deliberately not a
WebRTC method: a frame is about 1.8 MiB at the default geometry, whereas the control channel has to
stay prompt. A recorder can therefore join camera observations to robot state on the board without
fighting the daemon or base64-encoding pixels.

What remains is publishing the detector's output on the robot state stream, so a behaviour can
consume it without opening a second observation channel. The detector already subscribes to the
raw branch and runs at a paced rate; publishing keeps perception next to the sensor, deriving
features rather than shipping pixels.

Once detections exist as state, the behaviours in `docs/ideas/autonomous_behavior.md` that currently
key on Bluetooth ("a duck is *nearby*") can key on sight ("a duck is *there*"): approaching,
Expand Down
19 changes: 11 additions & 8 deletions docs/project/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -252,17 +252,20 @@ version line.

**What is missing is at the two ends, not in the middle:**

- **`robotd` cannot reload.** There is no SIGHUP handler and no way to swap an `ort` session
under a running 50 Hz loop. This is the milestone's real engineering: the swap must not drop a
tick, and a model whose shape is not `obs[1,61] → actions[1,14]` has to be refused *before* it
goes live rather than at the first inference.
- **Nothing publishes a bundle.** `xtask package --channel model-walk` is close — it checks
`--version` against the crate version, which a model does not have — and the HF repo layout
and naming do not exist.
- **Safe reload is implemented, but needs board evidence.** `robotd` coalesces SIGHUP requests,
waits until policy control is disabled, builds the candidate off the 50 Hz loop, and keeps the
current controller if loading fails. The remaining proof is a board run measuring the reload
boundary and refusing a malformed ONNX model before it is used.
- **A bundle can now be made, but nothing publishes one.** `cargo xtask package --channel
model-walk --model-dir <dir> --model-api 1 --version <version>` makes an independently-versioned
artifact whose manifest carries its compatibility API; it does not accept daemon hooks or
binaries. The HF repo layout, release workflow and naming still do not exist.
- **A third signing key.** `release-1` is CI's and `team.dev` installs nothing on a customer
robot, so *who may publish a policy a robot will run* is a new custody question, not a reuse
of an existing one.
- **`model_api`** (§5.5) is designed and unimplemented on both sides.
- **`model_api`** (§5.5) is checked by the updater against the running daemon and is required
when packaging a model bundle. It still needs a real published model to exercise that boundary
on a board.
- **The training loop.** `microduck_rl` trains and exports to ONNX; nothing carries the result
to a board without a daemon release. The model equivalent of `dev-push.sh` is what makes
"train it and try it" a minute rather than a CI run.
Expand Down
13 changes: 12 additions & 1 deletion docs/robot/cheatsheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,18 @@ Hardware and software in one report. Exits non-zero when the robot is unhealthy
it can gate a script — a hot motor or a pinned component is reported, not judged, and does not
affect the exit code. `--json` for a support bundle.

### A support bundle

```
robotctl support
```

Writes `/var/tmp/microduck-support.txt`: health and version reports, relevant unit status, the
latest daemon journal, and update history. It does not change the robot. Lines that may carry a
credential are removed before the file is written, but review the file before sharing it because
it still identifies the robot and describes its software state. Choose another destination with
`--output path/to/report.txt`.

### Watching the loop

```
Expand Down Expand Up @@ -790,4 +802,3 @@ eval "$(robotctl completions bash)"
```

`zsh`, `fish`, `elvish` and `powershell` work in place of `bash`.

47 changes: 47 additions & 0 deletions docs/robot/lerobot-local-recording.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Local LeRobot recording

Before hardware arrives, run `bash scripts/record-lerobot-preflight.sh`. It uses synthetic data only.

On a robot, record one 30-second local episode, validate it, then inspect the report. Do not export
or train on an episode with missing frames, non-monotonic capture times, or malformed actions.
The recorder never sends a control command and never uploads data. Images remain under
`/var/lib/robot/datasets/` until an operator deliberately copies them for local development.

Recording stops at the first configured budget: 300 frames, 512 MiB of raw images, or a 1 GiB
free-space reserve by default. It also refuses a 21st local episode until the existing ones have
been reviewed or archived. These are deliberate experiment-cost guardrails, not a retention
policy: choose explicit `--max-*` values for a larger, reviewed run. The validator rejects a
dataset that exceeds its recorded frame or byte budget.

Before converting, run the exporter's `--dry-run`. It reports policy-labelled frames and the
estimated uncompressed RGB footprint, refusing more than 300 frames or 1 GiB by default. Creating
the local LeRobot dataset then requires `--confirm-export`, and its destination must be empty:

```
python3 scripts/export-lerobot-local.py /var/lib/robot/datasets/<episode> \
--root /var/lib/robot/lerobot/<episode> --dry-run
python3 scripts/export-lerobot-local.py /var/lib/robot/datasets/<episode> \
--root /var/lib/robot/lerobot/<episode> --confirm-export
```

## Before trying a model update

With policy control disabled, run a read-only update dry run. It refuses an armed policy, a fallen
or limp robot, an unhealthy daemon, and an older daemon that cannot report its policy state. The
dry run then uses the normal updater path to verify the candidate's signature, hash and
`model_api`, but stops before moving `current` or signalling `robotd`:

```
python3 scripts/model-update-preflight.py model-walk
# for a signed local artifact directory:
python3 scripts/model-update-preflight.py model-walk --from /path/to/signed-artifacts
```

After an operator applies the model through the normal updater flow, observe the disabled-policy
control loop for ten seconds before re-enabling policy control. This command does not signal,
reload, or change robot settings; it writes a local report and rejects a policy-enabled, fallen,
limp, unhealthy, or missed-tick observation:

```
python3 scripts/observe-model-reload.py
```
54 changes: 54 additions & 0 deletions docs/robot/model-channel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Model channel contract

A policy is a separately versioned `model-*` component. It is not a daemon release: it carries
weights and model metadata only, and it cannot carry daemon binaries or install hooks. This keeps
a model trial inside the updater's ordinary signature, hash, compatibility, rollback and pinning
boundaries without giving a policy artifact a second way to alter a robot.

## Bundle

Build a bundle from a directory containing the files for one named policy slot. Files are installed
at the root of that model component's release, so a configured slot can name, for example,
`/opt/robot/model/walk/current/walk.onnx` directly.

```
cargo xtask package \
--channel model-walk \
--version 1.2.3 \
--model-dir /path/to/walk-bundle \
--model-api 1 \
--out dist/
```

`--model-api` is required. A robot accepts the artifact only when its running daemon implements at
least that API. The package command rejects non-`model-*` channels, daemon hooks, and daemon binary
layout for model bundles. Sign the resulting artifact and manifests with a policy signing key whose
public half is already trusted by the target robot; never put a private key or passphrase in this
repository, a command history, or a model bundle.

The bundle is deliberately shallow: package the ONNX file and the small metadata needed to run it,
not recordings, checkpoints, training logs, notebooks, or dependencies. Those belong to the
training environment, not a robot update.

## Trial sequence

1. Disable policy control and leave the robot upright, not limp or fallen.
2. Run `python3 scripts/model-update-preflight.py model-walk` (add `--from` for a signed local
artifact directory). It verifies the stopped state, health, signature, hash and compatibility
through the updater's dry-run path; it does not swap `current`.
3. An operator applies the configured model component with the normal `robotctl update apply`
workflow. This is the only step that may move `current` or request a reload.
4. Before re-enabling policy control, run `python3 scripts/observe-model-reload.py`. It records a
ten-second disabled-policy window and rejects missed ticks, a safety event, or unhealthy state.
5. Inspect the local observation report and update transcript. Only then make a deliberate,
supervised decision to re-enable policy control.

An older daemon that does not report `policy_enabled` fails step 2 rather than being assumed safe.
An incompatible or malformed model must remain non-current; do not bypass the preflight by copying
files into a component's `current` path.

## Experiment budget

Use short, local-only recordings for the first trial. The recorder and converter default to bounded
frames, storage, episode count and export size; see [local LeRobot recording](lerobot-local-recording.md).
Increase a `--max-*` limit only for a reviewed run with a stated metric and a stopping condition.
73 changes: 73 additions & 0 deletions duck-ipc-proto/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,10 @@ pub mod socket {
/// Under `/run/tofd/` for the same reason as the pad's: it is that unit's
/// `RuntimeDirectory=`, so systemd removes the socket when the daemon stops.
pub const TOF: &str = "/run/tofd/tof.sock";

/// `mediad`'s on-demand raw-frame endpoint. It is local-only: a raw camera frame is for a
/// recorder or perception process on the robot, not a multi-megabyte WebRTC control reply.
pub const MEDIA: &str = "/run/mediad/media.sock";
}

/// Where each daemon publishes what it is running: `/run/<service>/identity.json`.
Expand Down Expand Up @@ -259,6 +263,10 @@ pub const JOINT_NAMES: [&str; 15] = [
pub mod method {
pub const HELLO: &str = "hello";

/// One raw camera frame. `mediad` answers the JSON-RPC header, followed immediately by the
/// bytes named in that header, on its local Unix socket.
pub const MEDIA_FRAME: &str = "media.frame";

pub const CHECK: &str = "update.check";
pub const APPLY: &str = "update.apply";
pub const ROLLBACK: &str = "update.rollback";
Expand Down Expand Up @@ -2566,13 +2574,24 @@ pub struct RobotState {
pub head: [f64; 4],
/// Which policy drove this tick: `walk`, `stand`, or `held` when none did.
pub policy: String,
/// Whether policy control is armed for this tick. `held` alone is not enough to decide that:
/// an enabled policy can be holding on a zero command. This lets an update preflight require
/// the disabled state before a model reload. Absent from older daemons, which callers must
/// treat as unknown rather than safe.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub policy_enabled: Option<bool>,
pub safety: SafetyState,
#[serde(rename = "loop")]
pub control_loop: LoopState,
/// Measured joint angles, radians, indexed as [`JOINT_NAMES`].
pub joints: Vec<f64>,
/// What was commanded, so a viewer can show tracking error rather than guessing at it.
pub targets: Vec<f64>,
/// Raw output of the policy which drove this tick, in its 14-wide policy order. It is the
/// action before scaling and filtering, so an imitation-learning recorder does not have to
/// reverse engineer an action from joint targets. Empty when no policy drove this tick.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub policy_action: Vec<f32>,
/// Where contact odometry believes the robot is. `default` so a frame from
/// a `robotd` predating the estimator still parses — zeros, like a robot
/// that has not moved.
Expand Down Expand Up @@ -4416,6 +4435,7 @@ mod tests {
},
head: [0.0; 4],
policy: "stand".into(),
policy_enabled: None,
safety: SafetyState {
fallen: false,
limp: false,
Expand All @@ -4428,6 +4448,7 @@ mod tests {
},
joints: vec![0.0; 15],
targets: vec![0.0; 15],
policy_action: vec![],
odom: OdomState::default(),
theremin: None,
chorale: None,
Expand Down Expand Up @@ -4473,6 +4494,7 @@ mod tests {
},
head: [0.0; 4],
policy: "walk".into(),
policy_enabled: Some(false),
safety: SafetyState {
fallen: false,
limp: false,
Expand All @@ -4485,6 +4507,7 @@ mod tests {
},
joints: vec![0.0; 15],
targets: vec![0.0; 15],
policy_action: vec![],
odom: OdomState::default(),
theremin: None,
chorale: None,
Expand All @@ -4493,6 +4516,7 @@ mod tests {
let line = serde_json::to_string(&Request::notify_state(&state)).unwrap();
assert!(line.contains(r#""method":"robot.state""#), "{line}");
assert!(line.contains(r#""move":"#), "{line}");
assert!(line.contains(r#""policy_enabled":false"#), "{line}");
assert!(line.contains(r#""loop":"#), "{line}");
assert!(!line.contains("movement"), "{line}");
assert!(!line.contains("control_loop"), "{line}");
Expand Down Expand Up @@ -4833,4 +4857,53 @@ mod tests {
released
);
}

#[test]
fn policy_action_is_present_only_for_a_policy_driven_tick() {
let state = RobotState {
t: 1.0,
movement: MoveState {
requested: [0.0; 3],
applied: [0.0; 3],
limited_by: vec![],
},
head: [0.0; 4],
policy: "walk".into(),
policy_enabled: None,
safety: SafetyState {
fallen: false,
limp: false,
gravity: [0.0, 0.0, -1.0],
gain: None,
},
control_loop: LoopState {
hz: 50.0,
missed: 0,
},
joints: vec![0.0; 15],
targets: vec![0.0; 15],
policy_action: vec![0.0; 14],
odom: OdomState::default(),
theremin: None,
chorale: None,
};
let line = serde_json::to_string(&state).unwrap();
assert!(line.contains("\"policy_action\":[0.0,0.0,0.0"), "{line}");
assert_eq!(
serde_json::from_str::<RobotState>(&line)
.unwrap()
.policy_action
.len(),
14
);
let held = RobotState {
policy_action: vec![],
..state
};
assert!(
!serde_json::to_string(&held)
.unwrap()
.contains("policy_action")
);
}
}
1 change: 1 addition & 0 deletions mediad/src/exposure.rs
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,7 @@ mod tests {
width: 1280,
height: 720,
format: CAPTURE_FORMAT,
captured_at: std::time::SystemTime::UNIX_EPOCH,
data,
}
}
Expand Down
Loading