simulation environment of multi node

[jiaduxie]

Do you have a recommended configuration tutorial for the multi node nest simulation environment? It can also be the brief steps of environment configuration and the required installation package.

[jarsi]

Hi jiaduxie,

could you provide a little bit more details about the problems you are running in?

The Readme.md of this repository has already some points on how to set up the environment and run the model.

The exact steps you have to take depend on the cluster you are using. What kind of job scheduler does it use? Is Python 3 available? Does it have MPI support? Have you already installed NEST?

A rough sketch:

1. Manually compile NEST on your cluster, make sure python and MPI is supported.
   Do not use the conda version (no MPI nor OpenMP support). Use an official release (NEST master has features which are not implemented in this repository yet). Depending on your cluster you need to load some packages (eg. Python, MPI...)

2. Make sure to install all python packages listed in requirements.txt.
   Run:
   pip3 install -r requirements.txt
   If the cluster does not allow this, try:
   pip3 install --user -r requirements.txt

3. Inform the job scheduler on your system and how to run the job
   You will need to copy the file config_template.py to config.py. Change base_path to the absolute path of the multi-area repository. Change data_path to the path where you want to store the output of the simulation. Adapt the jobscript_template to your system. If it is SLURM, then there is already an example in place which you will could uncomment and try to use. Make sure to also load all packages that a multi-node simulation requires (eg. MPI). Change submit_cmd from None to the command your job scheduler uses. For SLURM it is sbatch.

4. Try to run run_example_fullscale.py
   Now you should be able to run
   python run_example_fullscale.py.
   This will set up the simulation environment, do all the preprocessing and finally submit the job to the cluster. Depending on your cluster you might want to change num_processes and local_num_threads.

If all of this works, you should be ready to run your own experiments.

I hope this helps!
Best,
Jari

[jiaduxie]

Oh,thinks Jari. I run the model in NEST of conda environment .But I installed the conda version of nest with MPI version，it also no？I try install NEST from source code,The MPI in NEST manually installed by myself conflict with the local MPI of the server?I'll ask you if you have any questions.

[jarsi]

I am only aware of a conda version of NEST which does not have MPI support. But maybe it exists.

To check whether your NEST version supports MPI and OpenMP, could you run in your environment the following command and post the output:

python -c "import nest; nest.Simulate(1.)"

My conda installed NEST gives me in the start_updating_ information that neither MPI nor OpenMP is available:

Sep 10 16:07:01 SimulationManager::start_updating_ [Info]:
Number of local nodes: 0
Simulation time (ms): 1
Not using OpenMP
Not using MPI

Concerning manual compilation. How did you try to compile NEST? Could you post what steps you have tried so far?

[jiaduxie]

I haven't started trying to compile manually.
I run the following in my conda environment,and the output is as follows:

$python -c "import nest; nest.Simulate(1.)"

Creating default RNGs
Creating new default global RNG
-- N E S T --
Copyright (C) 2004 The NEST Initiative
Version: nest-2.18.0
Built: Jan 27 2020 12:49:17

This program is provided AS IS and comes with
NO WARRANTY. See the file LICENSE for details.

Problems or suggestions?
Visit https://www.nest-simulator.org
Type 'nest.help()' to find out more about NEST.

Sep 10 22:20:26 NodeManager::prepare_nodes [Info]:
Preparing 0 nodes for simulation.

Sep 10 22:20:26 SimulationManager::start_updating_ [Info]:
Number of local nodes: 0
Simulation time (ms): 1
Number of OpenMP threads: 1
Number of MPI processes: 1

Sep 10 22:20:26 SimulationManager::run [Info]:
Simulation finished.

[jarsi]

It seems alright. Have you installed the packages from requirements.txt? Have you tired running a simulation?

[jiaduxie]

Yes,I have installed the packages from requirements.txt?Can you help me see if the command to execute multi-node simulation is like this?:
mpirun -hostfile hostfile python run_example_downscaled.py

The hostfile is following：
work0 slots = 2
work1 slots = 2

[jarsi]

I have no experience with hostfiles, but it looks reasonable to me. Have you adjusted num_processes and local_num_threads in the sim_dict? Have you tried running it? Did it work?

The run_example_downscaled.py is meant to be run on a local machine, for example a laptop. If you would like to experiment on a compute cluster you should exchange M.simulation.simulate() with start_job(M.simulation.label, submit_cmd, jobscript_template) (see run_example_fullscale.py) and additionally import:

from start_jobs import start_job
from config import submit_cmd, jobscript_template

In this case you need to invoke the script serially:

python run_example.py

The parallelized part is then specified in the jobscript_template in config.py.

[jiaduxie]

Hei,jarsi .If run a complete model on a cluster of two servers, about how much memory each machine needs to support？

[jarsi]

The model consumes approximately 1 TB of memory. So with two servers each server would need to provide 500 GB.

[jiaduxie]

Okay, thank you. Then,when you run the entire model, you use several servers and how much memory each is.

[jiaduxie]

Hei,jarsi.In what system are you running multiple nodes in parallel.My system is ubuntu, slurm configuration is not good.Do you have any guidance on configuring the environment?

[jarsi]

Hi, we do not set up the systems ourselves. We use for example JURECA from the Forschungszentrum Juelich It has everything we need already installed. What kind of system are you using?

[jiaduxie]

I am a server under linux system, the release version is ubuntu.In addition to running under JURECA, do you have your own running on a general server?

[jiaduxie]

Hi,jarsi,I am now simulating a small network for testing on two machines, and run it with the following command. It seems that the two machines run by themselves without interaction.

`mpirun.mpich -np 2 -host work0,work1 python ./multi_test.py`

In addition，Have you run his model of multi-area-model in your own cluster environment?