Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,9 @@
# executorlib with SLURM
A number of example for applying [executorlib](https://executorlib.readthedocs.io/) on [CMTI](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html).
A number of examples for applying [executorlib](https://executorlib.readthedocs.io/) on [CMTI](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html).

## Table of Contents
* [Getting Started](https://pyiron-dev.github.io/executorlib-with-slurm/slurm_demo_v1.html) - submit a serial function, an MPI-parallel function, and nested executors via SLURM
* [Submission Errors](https://pyiron-dev.github.io/executorlib-with-slurm/slurm_error_v1.html) - inspect `sbatch` submission failures raised through a `Future`
* [SLURM with Flux](https://pyiron-dev.github.io/executorlib-with-slurm/slurm_with_flux_v1.html) - run a nested [Flux](https://flux-framework.org) instance inside a SLURM allocation
* [SLURM with LAMMPS](https://pyiron-dev.github.io/executorlib-with-slurm/slurm_with_lammps_v1.html) - couple executorlib with [pylammpsmpi](https://pylammpsmpi.readthedocs.io) for interactive MPI-parallel LAMMPS
* [Nested SLURM Jobs](https://pyiron-dev.github.io/executorlib-with-slurm/slurm_with_nested_executor.html) - fan out many short tasks as job steps inside one SLURM allocation
2 changes: 1 addition & 1 deletion slurm_demo_v1.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"id": "c1177617-771e-46bb-ae67-f5bf6d7b914f",
"metadata": {},
"source": "# executorlib with SLURM \n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster, without having to rewrite the functions themselves. This notebook walks through submitting a serial function, an MPI-parallel function, and nested executors via SLURM.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials."
"source": "# Getting Started\n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster, without having to rewrite the functions themselves. This notebook walks through submitting a serial function, an MPI-parallel function, and nested executors via SLURM.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials."
},
{
"cell_type": "code",
Expand Down
2 changes: 1 addition & 1 deletion slurm_error_v1.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"id": "c1177617-771e-46bb-ae67-f5bf6d7b914f",
"metadata": {},
"source": "# executorlib with SLURM \n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook deliberately submits a job with an incomplete `resource_dict` to show how `SlurmClusterExecutor` surfaces `sbatch` submission failures through the returned `Future`. See the [resource dictionary troubleshooting page](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#resource-dictionary) for the required keys.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials."
"source": "# Submission Errors\n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook deliberately submits a job with an incomplete `resource_dict` to show how `SlurmClusterExecutor` surfaces `sbatch` submission failures through the returned `Future`. See the [resource dictionary troubleshooting page](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#resource-dictionary) for the required keys.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials."
},
{
"cell_type": "code",
Expand Down
2 changes: 1 addition & 1 deletion slurm_with_flux_v1.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"id": "c1177617-771e-46bb-ae67-f5bf6d7b914f",
"metadata": {},
"source": "# executorlib with SLURM \n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook shows how to run a [Flux](https://flux-framework.org) instance nested inside a SLURM allocation, see the [Flux section of the HPC Cluster Executor documentation](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux) for background.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials."
"source": "# SLURM with Flux\n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook shows how to run a [Flux](https://flux-framework.org) instance nested inside a SLURM allocation, see the [Flux section of the HPC Cluster Executor documentation](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux) for background.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials."
},
{
"cell_type": "markdown",
Expand Down
2 changes: 1 addition & 1 deletion slurm_with_lammps_v1.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
{
"cell_type": "markdown",
"id": "595b7bbc",
"source": "# executorlib with SLURM and LAMMPS\n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook couples executorlib with [pylammpsmpi](https://pylammpsmpi.readthedocs.io) to run an interactive, MPI-parallel [LAMMPS](https://www.lammps.org) molecular dynamics simulation inside a single persistent SLURM allocation. See the [pylammpsmpi integration guide](https://executorlib.readthedocs.io/en/latest/coupling.html#pylammpsmpi) for background.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials.",
"source": "# SLURM with LAMMPS\n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook couples executorlib with [pylammpsmpi](https://pylammpsmpi.readthedocs.io) to run an interactive, MPI-parallel [LAMMPS](https://www.lammps.org) molecular dynamics simulation inside a single persistent SLURM allocation. See the [pylammpsmpi integration guide](https://executorlib.readthedocs.io/en/latest/coupling.html#pylammpsmpi) for background.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials.",
"metadata": {}
},
{
Expand Down
2 changes: 1 addition & 1 deletion slurm_with_nested_executor.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
{
"cell_type": "markdown",
"id": "2148ffa0",
"source": "# executorlib with SLURM: Nested Executors\n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook submits a single outer SLURM job via `SlurmClusterExecutor`, which itself opens a second, nested `SlurmJobExecutor` to fan out many short-lived tasks as job steps within that same allocation. See the [HPC Job Executor documentation](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html) for details on nesting executors.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials.",
"source": "# Nested SLURM Jobs\n[executorlib](https://executorlib.readthedocs.io) extends the `Executor` interface from the Python standard library's [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) to distribute Python functions as jobs on an HPC cluster. This notebook submits a single outer SLURM job via `SlurmClusterExecutor`, which itself opens a second, nested `SlurmJobExecutor` to fan out many short-lived tasks as job steps within that same allocation. See the [HPC Job Executor documentation](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html) for details on nesting executors.\n\nBased on the [cmti and cmmg clusters](https://docs.mpcdf.mpg.de/doc/computing/clusters/systems/Sustainable_Materials.html) hosted at the MPCDF for the MPI for Sustainable Materials.",
"metadata": {}
},
{
Expand Down
Loading