Opentrons Buildroot Overlays

These are the Opentrons package and config overlays for Buildroot for building the OT-2.

(Highly) Recommended Reading

Working in this repository is easier if you are familiar with Buildroot.

• Documentation and users manual: https://buildroot.org/docs.html
• Upstream repository: https://github.com/buildroot/buildroot/
• Buildroot website: https://buildroot.org/

What is this repo?

This repository is a buildroot external tree containing board configuration files; package definitions version overrides; and CI build setup and code.

Building Locally

We wrap the Buildroot build system in a Docker container to manage dependencies and make it slightly more portable. Due to case-sensitive filenames and other concerns, the build must be run on Linux.

This fork requires the opentrons repo to be checked out next to it, since it uses a Buildroot external tree to build in our dependencies. Since this repo itself is also a Buildroot external tree, we need actual buildroot cloned next to it in a directory called buildroot-upstream (this can be overridden). The version of buildroot that is used in this repo is in buildroot-upstream-ref.

Prerequisites

• Linux
  • If you are on another OS, you should use a local or cloud Linux virtual machine
  • A tool like Vagrant can help with VM configuration and setup
• docker
• git

Setup

In order to build, docker and git must be installed. The monorepo must be checked out as a neighbor to buildroot. The upstream buildroot repo must be checked out as a neighbor to this repo at buildroot-upstream.

# clone the opentrons monorepo, if you haven't already
git clone https://github.com/Opentrons/opentrons.git

# clone the overlays next to the monorepo
git clone https://github.com/Opentrons/buildroot.git

# clone the upstream buildroot next to the overlays using a helper script that will run
# git clone https://github.com/buildroot/buildroot.git --branch=<current version> --depth=0 buildroot-upstream
# where <current version> is what version of upstream we use right now.
cd buildroot
./checkout-buildroot.sh

The checkout-buildroot script reads ./buildroot-up stream-ref and pulls the repo at https://github.com/buildroot/buildroot to that ref.

Build

From buildroot, run the ./opentrons-build.sh script. This will build a docker container and run a build in it using the Buildroot Makefile.

# build all images
./opentrons-build.sh

# equivalent command
./opentrons-build.sh all

Buildroot caches build intermediates aggressively, but sometimes you need to run a full rebuild. Full rebuilds (as well as your very first builds) can take a long time, so avoid them if you can. The Buildroot docs explain when a full rebuild is necessary.

# run a full rebuild
./opentrons-build.sh "clean all"

There's also a handy bind that gets you a terminal in the container:

# get a terminal
./opentrons-build.sh /bin/bash

Outputs

The outputs of the build are

• output/images/ot2-system.zip,
  • System image file for doing a software update
  • Contains a rootfs image that will be written to the unused root partition block device, and therefore can only change things mounted in on-disk directories other than /var
  • If your build is signed, there will also be signature files in output/images
  • If your build is unsigned you will need to disable signature checking in the robot's update server config (/var/lib/otupdate/config.json) and restart it (systemctl restart opentrons-update-server).
• output/images/ot2-fullimage.zip
  • Full image file for burning to a blank SD card
  • Contrasted to ot2-system.zip, the full image has a disk image file that contains:
    • The boot parittion
    • Two copies of the root partition
    • A small varfs partition
    • A partition table that carves out extra space for varfs
  • When a freshly-flashed SD card boots, a systemd unit runs to expand the varfs partition to take all available space on the SD card, which can take a long time
• output/images/VERSION.json
  • Version file for looking at the versions in the build

Other Makefile Targets

./opentrons-build.sh will pass the last argument to the Buildroot Makefile and any preceding arguments, if they exist, to docker run. So, if you want to run multiple make targets at once, wrap your targets in quotation marks:

./opentrons-build.sh "python-opentrons-api all"

Certain tasks require terminal input and output. For those tasks, you should run docker in interactive TTY mode to use your terminal inside the container:

# pass "-it" to `docker run` and "menuconfig" to `make`
./opentrons-build.sh -it menuconfig

Production Builds

You can control the release type with the OT_BUILD_TYPE environment variable. If you set this to release, the system will try and sign the output using a private key in the SIGNING_KEY environment variable. Opentrons production builds are never run locally; the CI is where production builds happen, and you will not have access to the production Opentrons signing key on your own machine. If you provide a SIGNING_KEY to the build, Opentrons OT-2 robots will refuse the update because the signature does not match the Opentrons public key.

# development build (default)
OT_BUILD_TYPE=dev ./opentrons-build.sh

# production build
OT_BUILD_TYPE=release SIGNING_KEY=super-secret ./opentrons-build.sh

The API key for our log aggregator, datadog, can be provided either by specifying it in the DATADOG_API_KEY environment variable or by having AWS credentials available to pull it (and python3 and boto3 installed). If not specified, the build will run and work, but that robot will be unable to upload logs to datadog.

# export data dog environment variable
export DATADOG_API_KEY=super-secret

# ...or pull datadog key from AWS credential store
pip3 install boto3
export AWS_ACCESS_KEY_ID=abc
export AWS_SECRET_ACCESS_KEY=xyz

Adding Commits

Commits and commit messages should stick to the conventions outlined in the Buildroot manual. Changes to the build configuration should be documented in a commit message in the form of configs/ot2_defconfig: select lib-foo. Changes to available build packages should receive a commit message in the form of package-name: (new,remove,update, etc.) package. This is different from our other repos because this is not a monorepo of unrelated packages; it is a single repo that does one thing (provide overlays for system builds), and we don't use conventional-commits to generate changelogs or anything.

Submitted pull requests should be made up of commits with messages that are suitable to be placed directly onto the head of opentrons-develop. Unlike the opentrons monorepo, we merge pull requests into this repository with a rebase merge instead of a squash merge. This means all commits will appear in the master branch once the PR is reviewed and accepted.

Modifying Configuration

We use the ot2 defconfig in configs/ot2_defconfig. This is where all the buildroot configuration should go. You can edit this file manually, but it is often easier to start by using the menuconfig tool.

Using menuconfig

Buildroot includes a TUI application for modifying the defconfig file. Please note that you should still inspect the defconfig after it is generated, and you may need to make manual edits before it is ready for merge.

# 1. load configs/ot2_defconfig into .config prior to editing
./opentrons-build.sh -it ot2_defconfig

# 2. open .config in the menuconfig tool
./opentrons-build.sh -it menuconfig

# 3. save the changes to .config back into configs/ot2_defconfig
./opentrons-build.sh -it savedefconfig

Buildroot also provides a menuconfig for editing linux kernel options, you need to build the project since the kernel makefiles are downloaded and live in output/build/linux-custom

# 1. open the kernel .config file in menuconfig
./opentrons-build.sh -it linux-menuconfig

# 2. save the changes to board/opentrons/ot2/kernel.config
./opentrons-build.sh -it linux-savedefconfig

Provisioning an SD Card and Booting

Once built, the file output/images/ot2-fullimage.zip can be flashed to a microSD card and used in an OT-2 using balenaEtcher.

1. Open balenaEtcher
2. Select path/to/buildroot/output/images/ot2-fullimage.zip
3. Select your SD card writer
4. Click "Flash!"
5. When done, insert the SD card into your Raspberry Pi and boot your OT-2
   • Due to how the filesystem is set up, the first boot can take a long time - on the order of half an hour
   • The button light will flash while the filesystem is being prepared

To gain SSH access to the device, connect the Pi to your computer via an Ethernet cable and:

1. Use the Opentrons App or the standalone Discovery Client to get the link-local IP address of the Pi
2. Follow the SSH setup guide to upload your SSH public key to the device

Note: development builds include a default SSH public key as part of the build located at board/opentrons/ot2/rootfs-overlay/var/home/.ssh/robot_key.pub.

Automated Builds and Deployment

We have integrations set up to build this repo in AWS Codebuild, which can build release or non-release builds. Release builds have the public cert for our signing key included, have signatures for the update files, and do not ship with an ssh public key already installed. Non-release builds do not have the signing key cert, do not have signatures, and ship with a default ssh public key for testing.

The repo is built automatically

• When any new object is pushed to this repo (non-release); this uses the current head of edge from the monorepo for the API server and update server
• When new commits are pushed to the edge branch of the monorepo (non-release), using the opentrons-develop branch of this repo
• When new commits are pushed to a branch of the monorepo that has a matching branch in this repo (non-release)
• When a tag is pushed to the monorepo (release), using the latest github release in this repo

When builds complete, they are uploaded to an s3 bucket prefixed by the codebuild build id (see the buildspec file). If the build was a release build, the manifest releases.json in the root of the bucket is updated so that the key path /production/{monorepo-version} contains a dict mapping system' to the path of the update file zip, 'fullImage' to the path of the provisioning zip, and 'version' to the path of the version file. For instance, the manifest might look like

{
  "production": {
    "3.8.3": {
      "fullImage": "https://opentrons-buildroot-ci.s3.amazonaws.com/ebc9f421-04db-4ec8-87bd-f990c69bbd80/opentrons-buildroot/ot2-fullimage.zip",
      "system": "https://opentrons-buildroot-ci.s3.amazonaws.com/ebc9f421-04db-4ec8-87bd-f990c69bbd80/opentrons-buildroot/ot2-system.zip",
      "version": "https://opentrons-buildroot-ci.s3.amazonaws.com/ebc9f421-04db-4ec8-87bd-f990c69bbd80/opentrons-buildroot/VERSION.json"
    }
  }
}