protocol-evaluation

FastAPI service for evaluating Opentrons protocols with asynchronous analysis plus simulation.

Features

• /info endpoint: Returns application version and protocol API to robot stack version mappings
• /ready endpoint: Readiness probe (API up + processor heartbeat seen recently)
• /evaluate endpoint: Accepts protocol files for analysis and simulation with optional custom labware, CSV data, and runtime parameters
• /jobs/{job_id}/status endpoint: Check the status of an evaluation job
• /jobs/{job_id}/result endpoint: Retrieve either the analysis or simulation artifact via the result_type query parameter
• Asynchronous processing: Evaluations run in a dedicated processor service

Architecture

This service uses a two-component architecture:

1. FastAPI Server (api/main.py): Handles file uploads and serves results
2. Processor Service (evaluate/processor.py): Runs analysis and simulation jobs asynchronously in the background

Jobs are queued via the filesystem at storage/jobs/{job_id}/ and the processor picks them up for evaluation.

Each job specifies a target robot server version. Supported versions range from 8.0.0 through the special next alias, which always points at the latest published Opentrons alpha build (configured in evaluate/env_config.py). The processor spins up isolated virtual environments (managed via uv) per version so evaluations stay reproducible.

In addition to release targets, the API also accepts an edge target intended for installing and analyzing against Opentrons' edge branch.

Development

Prerequisites

• Python >= 3.10
• uv for dependency management

Setup

# Install dependencies
make setup

# Run linter (check only, no fixes)
make lint

# Run tests (unit + integration)
make test

# Run end-to-end tests (starts services automatically)
make test-e2e

# Run all tests and linting
make test-all

# Format code
make format

CI/CD

The project uses GitHub Actions for continuous integration:

• Linting: Runs ruff to check code quality
• Unit & Integration Tests: Fast tests without services
• End-to-End Tests: Full workflow tests with services running

All checks run automatically on pull requests and pushes to main.

TODO – RTP overrides: Runtime parameter (RTP) override scenarios are not yet implemented or tested end-to-end. Once someone needs RTP overrides, we can extend the processor/API/tests to cover the behavior.

Running the Services

Start the FastAPI server:

make run-api
# Or manually:
uv run fastapi dev api/main.py

Start the processor service (in a separate terminal):

make run-processor
# Or manually:
uv run python run_processor.py

For one-shot processing (process pending jobs and exit):

make run-processor-once

The API will be available at http://localhost:8000

API Documentation

Once the server is running, you can access:

• Interactive API docs: http://localhost:8000/docs
• ReDoc documentation: http://localhost:8000/redoc

Testing

Run all tests:

make test

Note: make test-e2e automatically starts both the API and processor services for you, so you don't need to run them manually before exercising full workflow tests.

Run only unit tests:

make test-unit

Run only integration tests:

make test-integration

Makefile Targets

• make setup - Install dependencies with uv (including dev tools)
• make teardown - Remove the project virtual environment
• make lint - Run ruff lint + format check (no fixes)
• make format - Run ruff check --fix and ruff format
• make test - Run unit + integration tests (excludes e2e)
• make test-unit / make test-integration - Run specific suites
• make test-e2e - Spin up both services, run E2E tests, capture logs
• make test-all - Run lint, fast tests, and e2e in sequence
• make run - Launch both API and processor in one terminal
• make run-api / make run-processor / make run-processor-once - Control individual services
• make run-client - Execute the example client workflow
• make clean-storage - Delete all queued job directories
• make clean-venvs - Remove analysis virtual environments (recreated on demand)
• make clean-e2e-artifacts - Remove e2e PID + log files (e2e-*.pid/.log)

Project Structure

protocol-evaluation/
├── api/
│   ├── __init__.py
│   ├── main.py              # FastAPI application and endpoints
│   ├── file_storage.py      # File storage service
│   ├── config.py            # Configuration
│   └── version_mapping.py   # Protocol API to robot stack version mappings
├── evaluate/
│   ├── __init__.py
│   ├── env_config.py        # Robot server environment definitions
│   ├── job_status.py        # Job status management helpers
│   ├── processor.py         # Analysis + simulation processor service
│   └── venv_manager.py      # Virtual environment lifecycle helpers
├── client/
│   ├── README.md            # Usage docs for EvaluationClient
│   └── evaluate_client.py   # Sync + async clients for the API
├── tests/
│   ├── unit/
│   │   ├── test_evaluate.py
│   │   ├── test_file_storage.py
│   │   ├── test_info.py
│   │   └── test_processor.py
│   ├── integration/
│   │   ├── test_evaluate.py
│   │   └── test_info.py
│   └── e2e/
│       └── test_protocol_analysis.py
├── run_processor.py         # CLI script for running processor
├── Makefile                 # Development tasks
├── pyproject.toml           # Project dependencies and configuration
└── README.md

Endpoints

GET /info

Returns application information including version and protocol API version mappings.

Response:

{
  "version": "0.1.0",
  "protocol_api_versions": {
    "2.0": "3.14.0",
    "2.1": "3.15.2",
    ...
    "2.26": "8.7.0"
  }
}

POST /evaluate

Accepts a protocol file for evaluation with optional custom labware, CSV data, and runtime parameters.

Parameters:

• protocol_file (required): Python protocol file (.py extension)
• labware_files (optional): Array of custom labware JSON files (.json extension)
• csv_file (optional): CSV data file (.csv or .txt extension)
• rtp (optional): Runtime parameters as JSON object
• robot_version (required): Target robot server version (e.g., 8.7.0, next)

Example using curl:

curl -X POST http://localhost:8000/evaluate \
  -F "robot_version=8.7.0" \
  -F "protocol_file=@my_protocol.py" \
  -F "labware_files=@custom_labware1.json" \
  -F "labware_files=@custom_labware2.json" \
  -F "csv_file=@plate_data.csv" \
  -F 'rtp={"volume": 100, "temperature": 37}'

Response:

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "protocol_file": "my_protocol.py",
  "labware_files": ["custom_labware1.json", "custom_labware2.json"],
  "csv_file": "plate_data.csv",
  "rtp": {
    "volume": 100,
    "temperature": 37
  },
  "robot_version": "8.7.0"
}

The job_id is a unique identifier for this evaluation job. Files are saved to storage/jobs/{job_id}/ with the following structure:

• {job_id}.py - The protocol file
• labware/ - Directory containing custom labware JSON files
• {original_name}.csv - Uploaded CSV/TXT file (if provided)
• status.json - Job status information
• completed_analysis.json - Result from opentrons.cli.analyze
• completed_simulation.json - Result (or skip reason) from opentrons.simulate

The RTP parameters are stored with the response but not persisted to disk.

Simulation is best-effort: if runtime parameter overrides or RTP CSV inputs are provided, the processor skips simulate and records the reason in completed_simulation.json.

GET /jobs/{job_id}/status

Check the status of an evaluation job.

Response:

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "updated_at": "2024-01-15T10:30:00.123456"
}

Status values: pending, processing, completed, failed

GET /jobs/{job_id}/result

Retrieve either the analysis or simulation results for a completed job. Use the optional result_type query parameter (analysis by default, or simulation).

Response:

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "result_type": "analysis",
  "result": {
    "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "status": "success",
    "files_analyzed": {
      "protocol_file": "my_protocol.py",
      "labware_files": ["custom_labware1.json"],
      "csv_file": "plate_data.csv"
    },
    "analysis": {
      "commands": [],
      "labware": [],
      "pipettes": [],
      "modules": [],
      "errors": [],
      "warnings": []
    },
    "metadata": {
      "protocol_api_version": "2.26",
      "processed_at": "2024-01-15T10:30:00Z"
    }
  }
}

To retrieve the simulation output instead, append `?result_type=simulation` to the request URL.

Job Processing Flow

1. Submit: Client POSTs to /evaluate with the protocol, optional labware/CSV, RTP payload, and robot version
2. Queue: API saves the files to storage/jobs/{job_id}/, persists metadata, and marks status as pending
3. Process: Processor picks up pending jobs, ensures the correct venv exists, runs analysis, then attempts simulation (skipping simulation when CSV/RTP overrides are present)
4. Complete: Processor writes completed_analysis.json, completed_simulation.json, and updates status to completed
5. Retrieve: Client GETs /jobs/{job_id}/result?result_type=analysis|simulation to fetch the desired artifact

The processor service can run:

• Daemon mode (default): Continuously polls for new jobs
• One-shot mode: Processes pending jobs and exits (useful for cron/scheduled tasks)

Usage Example

Here's a complete example workflow:

1. Start the Services

Terminal 1 - Start the API server:

make run-api

Terminal 2 - Start the processor:

make run-processor

2. Submit a Protocol for Evaluation

curl -X POST http://localhost:8000/evaluate \
  -F "robot_version=8.7.0" \
  -F "protocol_file=@example_protocol.py" \
  | jq '.'

Response includes the assigned job_id, recorded filenames, optional RTP payload, and robot version.

3. Check Job Status

curl http://localhost:8000/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/status | jq '.'

Response:

{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "completed",
  "updated_at": "2024-01-15T10:30:00.123456"
}

4. Retrieve Evaluation Results

# Analysis output (default)
curl "http://localhost:8000/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/result" | jq '.'

# Simulation output
curl "http://localhost:8000/jobs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/result?result_type=simulation" | jq '.'

The analysis response contains protocol metadata, commands, and warnings. The simulation response mirrors completed_simulation.json and may indicate status: "skipped" with a reason when RTP overrides or CSV inputs prevent running simulate.