Sophios

Sophios is a workflow compiler. You write a workflow in Python, or in .wic, a short YAML format. Sophios holds it in a workflow model of its own, checks all of it before anything runs, and compiles it for a backend. Today that backend is the Common Workflow Language (CWL), so what it writes runs under any CWL runner, with or without Sophios installed: cwltool on your machine, or Toil, which sends the steps to a cloud or to Slurm, LSF and other cluster schedulers.

An analysis is rarely one program: you clean the raw data, fit a model and compute statistics on it side by side, and put both into a figure. Steps, and the data that flows between them, make a graph. Run by hand or from a shell script, that graph works on your machine today; Sophios makes it a workflow that runs the same way on next month's data, on a colleague's machine and on a cluster. You describe each step: which program it runs, what you give it and which earlier results it reads. Sophios fills in the connections you leave out and tells you what it chose, stops on a value of the wrong type, a missing input file or a container engine that is not running, and runs every program in the container it asks for.

Sophios's Python API has two halves. sophios.api.python.workflow builds workflows from Step and Workflow objects. sophios.api.python.tool_builder builds the tools themselves, so a program of your own becomes a step without hand-written CWL.

Documentation

The documentation is on Read the Docs. Start with Install, then Your first workflow.

Install

1. Install Docker and start it. Sophios runs each tool in the container image it names.

2. Create a conda environment with the programs pip cannot install, Node.js (the newest 26.x) and Graphviz:

   conda create --name sophios --channel conda-forge python=3.11 nodejs=26 graphviz
   conda activate sophios

3. Install Sophios, and let it write its configuration and copy the tools that ship with it to ~/wic/:

   python -m pip install sophios
   sophios --generate_config

4. Check that it works:

   mkdir sophios-check && cd sophios-check
   sophios --yaml ~/wic/examples/multistep1.wic --run_local --quiet

   The last line starts with Success! Runner outputs are under.

The install guide explains each step and what to do when one fails. To work on Sophios itself, start with the developer guide.

A first workflow

This workflow makes an empty file, writes a greeting into it, and reads the file back. In a project directory, save either form:

# greeting.wic
steps:
- id: touch
  in:
    filename: !ii greeting.txt
  out:
  - file: !& empty_file
- id: append
  in:
    file: !* empty_file
    str: !ii Hello, Sophios
  out:
  - file: !& greeting_file
- id: cat
  in:
    file: !* greeting_file

# greeting.py
"""Create a file, write a greeting into it, and read it back."""
from pathlib import Path

from sophios.api.python.workflow import Step, Workflow


#: The tools that ship with Sophios, where `sophios --generate_config` put them.
ADAPTERS = Path.home() / 'wic' / 'cwl_adapters'


def workflow() -> Workflow:
    """The same three steps as greeting.wic."""
    touch = Step(clt_path=ADAPTERS / 'touch.cwl')
    touch.inputs.filename = 'greeting.txt'
    append = Step(clt_path=ADAPTERS / 'append.cwl')
    append.inputs.file = touch.outputs.file
    append.inputs.str = 'Hello, Sophios'
    cat = Step(clt_path=ADAPTERS / 'cat.cwl')
    cat.inputs.file = append.outputs.file
    return Workflow([touch, append, cat], 'greeting')


if __name__ == '__main__':
    workflow().run()

!ii marks a value written into the workflow; !& names a step's output and !* feeds it into a later step's input. In Python, an assignment does each, and ADAPTERS is where sophios --generate_config put the tools that ship with Sophios. Run either form:

sophios --yaml greeting.wic --run_local
python greeting.py

Both compile to the same CWL, check the machine, fetch the container images and run. Sophios writes the compiled CWL, the job file and the results under autogenerated/ in the directory you run from. --basepath (or basepath= in Python) lets you choose another directory for them, and --outdir one for only the files the run produces. Every problem Sophios finds carries a code, and sophios --explain <code> says what to change.

Writing a tool

When a program has no CWL description yet, the tool builder writes one, and the tool goes straight into a workflow. Here wc -w counts the words of the greeting:

# count_words.py
"""Build a tool in Python, then run it in a workflow."""
from pathlib import Path

from sophios.api.python.tool_builder import (
    CommandLineTool, Input, Inputs, Output, Outputs, cwl,
)
from sophios.api.python.workflow import Step, Workflow

#: The tools that ship with Sophios, where `sophios --generate_config` put them.
ADAPTERS = Path.home() / 'wic' / 'cwl_adapters'

word_count = (
    CommandLineTool('word_count',
                    Inputs(text=Input(cwl.file, position=1)),
                    Outputs(counts=Output(cwl.file, glob='counts.txt')))
    .base_command('wc', '-w')
    .stdout('counts.txt')
    .docker('debian:bookworm-slim')
)


def workflow() -> Workflow:
    """Create a file, write a line into it, and count its words."""
    touch = Step(clt_path=ADAPTERS / 'touch.cwl')
    touch.inputs.filename = 'greeting.txt'
    append = Step(clt_path=ADAPTERS / 'append.cwl')
    append.inputs.file = touch.outputs.file
    append.inputs.str = 'Hello, Sophios'
    count = Step(word_count)
    count.inputs.text = append.outputs.file
    return Workflow([touch, append, count], 'count_words')


if __name__ == '__main__':
    word_count.write_cwl('word_count.cwl', validate=True)
    workflow().run()

The run writes word_count.cwl, a tool file any .wic workflow can use, and leaves counts.txt in the output directory. Writing tools in Python takes the builder one part at a time.

License

MIT; see LICENSE.