Fuzzball Executor Plugin for Nextflow

The following repository contains source code for a Nextflow plugin implementing support for CIQ's product Fuzzball. This plugin enables use of high quality, community maintained workflows provided by the nf-core on Fuzzball.

Overview

The nf-fuzzball-submit tool is provided to submit a parent workflow that starts the Nextflow workflow in a Fuzzball deployment. The parent workflow performs the following tasks to set up the environment to submit child Nextflow jobs to Fuzzball:

1. Sets up the Fuzzball executor plugin in the parent workflow through Fuzzball file ingress and the setup job.
2. Authenticates with Fuzzball (via config file, direct login, or device flow) and passes credentials as a Fuzzball secret so the plugin can renew tokens during long-running pipelines. This is handled in the setup job of the submitted parent workflow.

When the Nextflow workflow is executed within Fuzzball using the Fuzzball executor, child jobs are translated into Fuzzball workflows (Fuzzfiles) and submitted to your Fuzzball deployment. If you are running on-premise, jobs should be scheduled on compute resources running Fuzzball Substrate that have available resources. If you are running in the cloud, instances will be provisioned to run Nextflow child jobs.

For more information on submission script usage, please see the submission script usage documentation.

Building and using development versions of the nextflow plugin

Build the plugin from the current working tree and push it to a location that is accessible via s3 or https from the Fuzzball cluster.

make push_dev

This depends on a script called push_dev.local you can create to customize where the plugin is pushed for use with your Fuzzball workflows. The plugin's name has to match the pattern the submission script looks for. Here is an example:

#! /bin/bash

ver="${1:-none}"
fb_ver="${2:=none}"
if [[ "${ver}${fb_ver}" =~ none ]] ; then
    echo "USAGE:  $0 PLUGIN_VERSION FUZZBALL_VERSION"
    echo "        PLUGIN_VERSION and FUZZBALL_VERSION are both expected to start with a v"
fi

aws s3 cp \
    build/distributions/nf-fuzzball-${ver}.zip \
    s3://MY_BUCKET/nf-fuzzball/${ver}/nf-fuzzball-${ver}-stable-${fb_ver}.zip

Note the naming pattern ${ver}/nf-fuzzball-${ver}-stable-${fb_ver}.zip where $ver is the full version of the plugin (e.g. 'v0.1.2') and $fb_ver is the MAJOR.MINOR version of fuzzball to build against (e.v. 'v2.2'). To use this build with the nf-fuzzball-submit script you would specify the location to download from like so:

nf-fuzzball-submit \
    --plugin-base-uri "s3://MY_BUCKET/MY_PREFIX/nf-fuzzball" \
    --s3-secret "secret://user/MY_S3_SECRET" \
    -- \
    nextflow run -profile fuzzball hello