docs: add explanatory markdown to example notebooks (#191)

The example notebooks under example_workflows/ had almost no prose,
only code cells, which made the documentation hard to follow for new
users. Add markdown cells explaining each workflow engine, what each
step of the notebook does, and how the "Load Workflow with X" sections
demonstrate interoperability, without changing any existing code cell.

Also expand documentation/evcurve.md and documentation/nfdi.md (which
were a few sentences each) to walk through the workflow.py functions
and workflow.json structure the same way documentation/arithmetic.md
already did, and lightly extend intro.md/conclusion.md.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Author: jan-janssen

documentation/conclusion.md

@@ -1,3 +1,17 @@
 # Conclusion 
 Based on the Python Workflow Definition three rather different workflows were implemented in rather different workflow 
-engines. This demonstrates the interoperability achieved with the Python Workflow Definition. 
+engines: a simple arithmetic example coupling two Python functions, a Quantum Espresso energy volume curve calculation 
+with fan-out parallelism, and a file based, multi-tool NFDI4Ing benchmark. In each case the same `workflow.json` was 
+written by one workflow engine and successfully loaded and executed by the others, without any change to the underlying 
+Python functions in `workflow.py`. This demonstrates the interoperability achieved with the Python Workflow Definition, 
+and shows that it scales from small, in-memory Python workflows to larger, file based, multi-environment scientific 
+workflows.
+
+## Where to go next
+* Browse the [example_workflows](https://github.com/pythonworkflow/python-workflow-definition/tree/main/example_workflows) 
+  directory for the full set of notebooks, including engines not covered in this book such as `pyiron_workflow`, 
+  `executorlib` and CWL.
+* See the [README](https://github.com/pythonworkflow/python-workflow-definition) for installation instructions via 
+  `pip` or `conda`.
+* Read the accompanying publication: [J. Janssen et al., A python workflow definition for computational materials 
+  design, Digital Discovery, 2025](https://doi.org/10.1039/D5DD00231A).

documentation/evcurve.md

@@ -1,4 +1,67 @@
 # Energy Volume Curve
 Based on [previous work](https://materialdigital.github.io/ADIS2023/README.html) from the [ADIS 2023 workshop](https://www.mpie.de/4902385/adis2023)
 the calculation of an energy volume curve with the [quantum espresso](https://www.quantum-espresso.org) density 
-functional theory (DFT) simulation code is implemented in the Python Workflow Definition.
+functional theory (DFT) simulation code is implemented in the Python Workflow Definition.
+
+## Workflow
+An energy-volume curve is computed by relaxing a bulk crystal structure, straining it to a series of volumes and
+computing the total energy at each volume with a self-consistent-field (SCF) calculation. The pipeline is implemented
+as five Python functions in [workflow.py](example_workflows/quantum_espresso/workflow.py):
+```python
+def get_bulk_structure(element, a, cubic):
+    # build an ASE bulk crystal structure, e.g. Al
+
+def calculate_qe(working_directory, input_dict):
+    # write a pw.x input file, run "pw.x -in input.pwi > output.pwo" and parse the pwscf.xml output;
+    # input_dict["calculation"] selects "vc-relax" (cell + geometry relaxation) or "scf" (single-point energy)
+
+def generate_structures(structure, strain_lst):
+    # apply each volumetric strain in strain_lst to a structure, returning one structure per strain
+
+def plot_energy_volume_curve(volume_lst, energy_lst):
+    # plot energy against volume and save it as evcurve.png
+```
+`write_input`/`collect_output` handle the file-based `pw.x` input/output (`.pwi`/`.pwo`/`.xml` files), and
+`ase_to_json`/`json_to_ase` (de)serialize ASE `Atoms` objects to/from OPTIMADE-JSON strings, so structures can be
+passed between functions as plain JSON data instead of Python objects.
+
+The workflow combines these functions as: `get_bulk_structure` -> `calculate_qe` (`vc-relax`, once) -> 
+`generate_structures` -> `calculate_qe` (`scf`, once per strained structure) -> `plot_energy_volume_curve`. With five
+strains this means one relaxation followed by five independent SCF calculations that fan out from
+`generate_structures` and feed back into a single plot.
+
+## workflow.json
+[workflow.json](example_workflows/quantum_espresso/workflow.json) encodes this fan-out as five separate
+`workflow.calculate_qe` function nodes, each connected to its own strained structure and its own `working_directory`
+input, all reading the same shared inputs (`pseudopotentials`, `kpts`, `calculation`, `smearing`) via
+`python_workflow_definition.shared.get_dict`. An excerpt showing the pattern for two of the five strained
+calculations:
+```
+{
+  "nodes": [
+    {"id": 2, "type": "function", "value": "workflow.generate_structures"},
+    {"id": 3, "type": "function", "value": "workflow.calculate_qe"},
+    {"id": 4, "type": "function", "value": "workflow.calculate_qe"},
+    {"id": 19, "type": "input", "value": "strain_0", "name": "working_directory_1"},
+    {"id": 20, "type": "function", "value": "python_workflow_definition.shared.get_dict"},
+    {"id": 22, "type": "input", "value": "strain_1", "name": "working_directory_2"},
+    {"id": 23, "type": "function", "value": "python_workflow_definition.shared.get_dict"}
+  ],
+  "edges": [
+    {"target": 3, "targetPort": "working_directory", "source": 19, "sourcePort": null},
+    {"target": 20, "targetPort": "structure", "source": 2, "sourcePort": "s_0"},
+    {"target": 3, "targetPort": "input_dict", "source": 20, "sourcePort": null},
+    {"target": 4, "targetPort": "working_directory", "source": 22, "sourcePort": null},
+    {"target": 23, "targetPort": "structure", "source": 2, "sourcePort": "s_1"},
+    {"target": 4, "targetPort": "input_dict", "source": 23, "sourcePort": null}
+  ]
+}
+```
+Node 2 (`generate_structures`) exposes one output port per strain (`s_0`, `s_1`, ...); each port is wired into its
+own `get_dict`/`calculate_qe` pair. The energy and volume outputs of all five `calculate_qe` nodes are collected back
+into lists with `python_workflow_definition.shared.get_list` before being passed to `plot_energy_volume_curve`.
+
+## Requirements
+Running this workflow (rather than just loading/inspecting the JSON) requires the Quantum Espresso `pw.x` binary,
+matching pseudopotential files (see [espresso/pseudo](example_workflows/quantum_espresso/espresso/pseudo)) and the
+Python dependencies listed in [environment.yml](example_workflows/quantum_espresso/environment.yml).

documentation/intro.md

@@ -13,6 +13,9 @@ Currently supported workflow engines:
 * [aiida-workgraph](https://github.com/aiidateam/aiida-workgraph)
 * [jobflow](https://github.com/materialsproject/jobflow)
 * [pyiron_base](https://github.com/pyiron/pyiron_base)
+* [pyiron_workflow](https://github.com/pyiron/pyiron_workflow)
+* [executorlib](https://github.com/pyiron/executorlib)
+* [CWL](https://www.commonwl.org/) via [cwltool](https://github.com/common-workflow-language/cwltool)
 
 ## Example Workflows
 Three workflows are implemented:

documentation/nfdi.md

@@ -1,4 +1,61 @@
 # NFDI4Ing Benchmark
-To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark 
-developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all three simulation codes 
-based on the Python Workflow Definition. 
+To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark
+developed as part of [NFDI4Ing](https://www.inggrid.org/article/id/3726/) is implemented for all workflow engines
+based on the Python Workflow Definition.
+
+## Workflow
+Unlike the [arithmetic](arithmetic.md) and energy-volume-curve examples, which pass Python objects between functions,
+this benchmark chains together external command line tools that each require their own conda environment (defined in
+`source/envs/*.yaml`) and communicate through files rather than in-memory values. The
+pipeline is defined by six Python functions in [workflow.py](example_workflows/nfdi/workflow.py), each of which
+copies its inputs into a stage-specific subdirectory (`preprocessing`, `processing` or `postprocessing`) and calls
+its external tool with `conda_subprocess.check_output`:
+```python
+def generate_mesh(domain_size: float, source_directory: str) -> str: ...
+def convert_to_xdmf(gmsh_output_file: str) -> dict: ...
+def poisson(meshio_output_xdmf: str, meshio_output_h5: str, source_directory: str) -> dict: ...
+def plot_over_line(poisson_output_pvd_file: str, poisson_output_vtu_file: str, source_directory: str) -> str: ...
+def substitute_macros(pvbatch_output_file: str, ndofs: int, domain_size: float, source_directory: str) -> str: ...
+def compile_paper(macros_tex: str, plot_file: str, source_directory: str) -> str: ...
+```
+* `generate_mesh` runs [`gmsh`](https://gmsh.info) on `unit_square.geo` to mesh a 2D square whose side length is set
+  by the `domain_size` parameter, producing a `.msh` file.
+* `convert_to_xdmf` runs `meshio convert` to turn the gmsh mesh into an XDMF/H5 file pair that the FEM solver can read.
+* `poisson` runs a [FEniCS](https://fenicsproject.org)-based `poisson.py` script that solves the Poisson equation on
+  the mesh, returning the number of degrees of freedom (`numdofs`) alongside the `.pvd`/`.vtu` result files.
+* `plot_over_line` runs ParaView's `pvbatch` with `postprocessing.py` to sample the FEM solution along a line and
+  write it out as a CSV file.
+* `substitute_macros` runs `prepare_paper_macros.py` to fill a LaTeX macro template with the plot data path, the
+  domain size and the number of degrees of freedom.
+* `compile_paper` runs [`tectonic`](https://tectonic-typesetting.github.io) to compile `paper.tex`, which references
+  those macros and the plot, into `paper.pdf`.
+
+The connection of these Python functions is stored in the [workflow.json](example_workflows/nfdi/workflow.json)
+JSON file, following the same `nodes`/`edges` structure as the other examples. Each function is a `function` node,
+`domain_size` and `source_directory` are `input` nodes, and edges connect a function's input port either to another
+function's output port or directly to an input node:
+```
+{
+  "version": "0.1.0",
+  "nodes": [
+    {"id": 0, "type": "function", "value": "workflow.generate_mesh"},
+    {"id": 1, "type": "function", "value": "workflow.convert_to_xdmf"},
+    {"id": 6, "type": "input", "value": 2.0, "name": "domain_size"},
+    {"id": 7, "type": "input", "value": "source", "name": "source_directory"},
+    {"id": 8, "type": "output", "name": "result"}
+  ],
+  "edges": [
+    {"target": 0, "targetPort": "domain_size", "source": 6, "sourcePort": null},
+    {"target": 0, "targetPort": "source_directory", "source": 7, "sourcePort": null},
+    {"target": 1, "targetPort": "gmsh_output_file", "source": 0, "sourcePort": null}
+  ]
+}
+```
+Since `convert_to_xdmf` and `poisson` return a `dict` instead of a single value, the edges that read their output
+set `sourcePort` to the specific dictionary key (e.g. `"xdmf_file"` or `"numdofs"`) rather than `null`.
+
+Because every stage shells out to a different external tool in a different conda environment and passes file paths
+along the graph, this benchmark exercises a different part of the Python Workflow Definition than the arithmetic and
+energy-volume-curve examples: it shows that the same `workflow.json` produced by one engine can be handed to another
+engine, an execution manager like [CWL](https://www.commonwl.org)/`cwltool`, or plain Python, and still reproduce the
+same chain of `gmsh`, `meshio`, FEniCS, `pvbatch` and `tectonic` calls end to end.

example_workflows/arithmetic/aiida.ipynb

@@ -23,12 +23,12 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "source": "# Aiida",
+   "source": "# Aiida\n\nThis notebook defines the arithmetic workflow with [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph) and then loads the resulting `workflow.json` into `jobflow`, `pyiron_base` and `pyiron_workflow`, to demonstrate that the same workflow definition can be executed by several different engines.",
    "metadata": {}
   },
   {
    "cell_type": "markdown",
-   "source": "## Define workflow with aiida",
+   "source": "## Define workflow with aiida\n\n`aiida-workgraph` represents a workflow as a `WorkGraph` of `Task`s. Tasks are executed and stored by the AiiDA engine, so we first need to connect to an AiiDA profile with `load_profile()` before any task can run.",
    "metadata": {}
   },
   {
@@ -63,6 +63,11 @@
    "outputs": [],
    "execution_count": 2
   },
+  {
+   "cell_type": "markdown",
+   "source": "The arithmetic functions are imported under a leading underscore because they are still plain Python functions at this point; `wg.add_task` below turns each one into an AiiDA `Task` that can be linked to other tasks' inputs and outputs. `get_prod_and_div` returns a dictionary, so it is wrapped with `task(outputs=['prod', 'div'])` to expose `prod` and `div` as separate output sockets.",
+   "metadata": {}
+  },
   {
    "cell_type": "code",
    "source": "wg = WorkGraph(\"arithmetic\")",
@@ -99,6 +104,11 @@
    "outputs": [],
    "execution_count": 6
   },
+  {
+   "cell_type": "markdown",
+   "source": "With all tasks connected, `write_workflow_json` walks the `WorkGraph`'s tasks and links and serializes them into the PWD `workflow.json` format: each task becomes a `function` node and each link becomes an edge between an input and an output port.",
+   "metadata": {}
+  },
   {
    "cell_type": "code",
    "source": "write_workflow_json(wg=wg, file_name=workflow_json_filename)",
@@ -125,7 +135,7 @@
   },
   {
    "cell_type": "markdown",
-   "source": "## Load Workflow with jobflow",
+   "source": "## Load Workflow with jobflow\n\nThe same `workflow.json` can now be loaded by a different engine. `python_workflow_definition.jobflow.load_workflow_json` reconstructs a jobflow `Flow` from the JSON, which is then executed locally with `run_locally`.",
    "metadata": {}
   },
   {
@@ -180,7 +190,7 @@
   },
   {
    "cell_type": "markdown",
-   "source": "## Load Workflow with pyiron_base",
+   "source": "## Load Workflow with pyiron_base\n\nThe same JSON file is loaded into `pyiron_base`, producing a list of delayed jobs. `.draw()` visualizes the dependency graph and `.pull()` triggers delayed execution, running each job in turn and returning the final result.",
    "metadata": {}
   },
   {
@@ -236,7 +246,7 @@
   {
    "metadata": {},
    "cell_type": "markdown",
-   "source": "## Load Workflow with pyiron_workflow"
+   "source": "## Load Workflow with pyiron_workflow\n\nFinally, the workflow is loaded into `pyiron_workflow`, producing a `Workflow` object whose node graph can be visualized with `.draw()` and executed with `.run()`."
   },
   {
    "metadata": {},
@@ -267,4 +277,4 @@
    "source": "wf.run()"
   }
  ]
-}
+}