EnTK hangs on Traverse when using multiple Nodes

[lsawade]

Hi,

I don't know whether this is related to #135 . It is weird because I got everything running on a single node, but as soon as I use more than one EnTK seems to hang. I checked out the submission script and it looks fine to me; so, did the node list.

The workflow already hangs in the submission of the first task, which is a single core, single thread task.

EnTK session: re.session.traverse.princeton.edu.lsawade.018666.0003
Creating AppManagerSetting up RabbitMQ system                                 ok
                                                                              ok
Validating and assigning resource manager                                     ok
Setting up RabbitMQ system                                                   n/a
new session: [re.session.traverse.princeton.edu.lsawade.018666.0003]           \
database   : [mongodb://specfm:****@129.114.17.185/specfm]                    ok
create pilot manager                                                          ok
submit 1 pilot(s)
        pilot.0000   princeton.traverse       90 cores      12 gpus           ok
All components created
create unit managerUpdate: pipeline.0000 state: SCHEDULING
Update: pipeline.0000.WriteSourcesStage state: SCHEDULING
Update: pipeline.0000.WriteSourcesStage.WriteSourcesTask state: SCHEDULING
Update: pipeline.0000.WriteSourcesStage.WriteSourcesTask state: SCHEDULED
Update: pipeline.0000.WriteSourcesStage state: SCHEDULED
MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
                                                           ok
submit: ########################################################################
Update: pipeline.0000.WriteSourcesStage.WriteSourcesTask state: SUBMITTING

[Ctrl + C]

close unit manager                                                            ok
...

Stack

  python               : /home/lsawade/.conda/envs/ve-entk/bin/python3
  pythonpath           : 
  version              : 3.8.2
  virtualenv           : ve-entk

  radical.entk         : 1.5.12-v1.5.12@HEAD-detached-at-v1.5.12
  radical.gtod         : 1.5.0
  radical.pilot        : 1.5.12
  radical.saga         : 1.5.9
  radical.utils        : 1.5.12

Client zip

client.session.zip

Session zip

sandbox.session.zip

[andre-merzky]

Hi @lsawade - this is a surprising one. The task stdout shows:

$ cat *err
srun: Job 126172 step creation temporarily disabled, retrying (Requested nodes are busy)

This one does look like a slurm problem. Is this reproducible?

[lsawade]

Reproduced! The message with step creation appears after a while. Meaning I continuously checked the task's error file, and eventually the message showed up!

[andre-merzky]

@lsawade , would you please open an ticket with Traverse support? Maybe our srun command is not well-formed for Traverse's Slurm installation? Please include the srun command:

/usr/bin/srun --exclusive --cpu-bind=none --nodes 1 --ntasks 1 --cpus-per-task 1 --gpus-per-task 0 --nodelist=/scratch/gpfs/lsawade/radical.pilot.sandbox/re.session.traverse.princeton.edu.lsawade.018666.0003/pilot.0000/unit.000000//unit.000000.nodes --export=ALL,NODE_LFS_PATH="/tmp" write-sources "-f" "/tigress/lsawade/entkdatabase/C200709121110A/C200709121110A.cmt" "-p" "/home/lsawade/gcmt3d/workflow/params" 

and the nodelist file which just contains:

traverse-k04g10

[lsawade]

It throws following error:

srun: error: Unable to create step for job 126202: Requested node configuration is not available

If I take out the nodelist argument, it runs

[andre-merzky]

Hmm, is that node name not valid somehow?

[lsawade]

I tried running it with the nodename as a string and that worked

/usr/bin/srun --nodelist=traverse-k05g10 --exclusive --cpu-bind=none --nodes 1 --ntasks 1 --cpus-per-task 1 --gpus-per-task 0 --export=ALL,NODE_LFS_PATH="/tmp" write-sources "-f" "/tigress/lsawade/entkdatabase/C200709121110A/C200709121110A.cmt" "-p" "/home/lsawade/gcmt3d/workflow/params"

---

Note that I'm using salloc and hence a different nodename

[lsawade]

I found the solution. When SLURM takes in a file for a nodelist, one has to use the node file option:

/usr/bin/srun --nodefile=nodelistfile --exclusive --cpu-bind=none --nodes 1 --ntasks 1 --cpus-per-task 1 --gpus-per-task 0 --export=ALL,NODE_LFS_PATH="/tmp" write-sources "-f" "/tigress/lsawade/entkdatabase/C200709121110A/C200709121110A.cmt" "-p" "/home/lsawade/gcmt3d/workflow/params"

[andre-merzky]

Oh! Thanks for tracking that down, we'll fix this!

[lsawade]

It is puzzling though, that srun doesn't throw an error. When I do it by hand, srun throws an error when feeding a nodelist file to the --nodelist= option

[andre-merzky]

@lsawade : the fix has been released, please let us know if that problem still happens!

[lsawade]

@andre-merzky, will test!

[lsawade]

Sorry, for the extraordinarily late feedback, but the issue seems to persist. It already hangs in the Hello, World task.
Did I update correctly?

---

My stack:

Details

  python               : /home/lsawade/.conda/envs/ve-entk/bin/python3
  pythonpath           : 
  version              : 3.8.2
  virtualenv           : ve-entk

  radical.entk         : 1.6.0
  radical.gtod         : 1.5.0
  radical.pilot        : 1.6.2
  radical.saga         : 1.6.1
  radical.utils        : 1.6.2

My script:

Details

from radical.entk import Pipeline, Stage, Task, AppManager
import traceback, sys, os


hostname = os.environ.get('RMQ_HOSTNAME', 'localhost')
port = int(os.environ.get('RMQ_PORT', 5672))
password = os.environ.get('RMQ_PASSWORD', None)
username = os.environ.get('RMQ_USERNAME', None)


specfem = "/scratch/gpfs/lsawade/MagicScripts/specfem3d_globe"

if __name__ == '__main__':
    p = Pipeline()

    # Hello World########################################################
    test_stage = Stage()
    test_stage.name = "HelloWorldStage"

    # Create 'Hello world' task
    t = Task()
    t.cpu_reqs = {'cpu_processes': 1, 'cpu_process_type': None, 'cpu_threads': 1, 'cpu_thread_type': None}
    t.pre_exec = ['module load openmpi/gcc']
    t.name = "HelloWorldTask"
    t.executable = '/bin/echo'
    t.arguments = ['Hello world!']
    t.download_output_data = ['STDOUT', 'STDERR']

    # Add task to stage and stage to pipeline
    test_stage.add_tasks(t)
    p.add_stages(test_stage)

    #########################################################    
    specfem_stage = Stage()
    specfem_stage.name = 'SimulationStage'
    
    for i in range(2):

        # Create Task
        t = Task()
        t.name = f"SIMULATION.{i}"
        tdir = f"/home/lsawade/simple_entk_specfem/specfem_run_{i}"
        t.pre_exec = [
            # Load necessary modules
            'module load openmpi/gcc',
            'module load cudatoolkit/11.0',
            
            # Change to your specfem run directory
            f'rm -rf {tdir}',
            f'mkdir {tdir}',
            f'cd {tdir}',
            
            # Create data structure in place
            f'ln -s {specfem}/bin .',
            f'ln -s {specfem}/DATABASES_MPI .',
            f'cp -r {specfem}/OUTPUT_FILES .',
            'mkdir DATA',
            f'cp {specfem}/DATA/CMTSOLUTION ./DATA/',
            f'cp {specfem}/DATA/STATIONS ./DATA/',
            f'cp {specfem}/DATA/Par_file ./DATA/'
        ]
        t.executable = './bin/xspecfem3D'
        t.cpu_reqs = {'cpu_processes': 4, 'cpu_process_type': 'MPI', 'cpu_threads': 1, 'cpu_thread_type' : 'OpenMP'}
        t.gpu_reqs = {'gpu_processes': 4, 'gpu_process_type': 'MPI', 'gpu_threads': 1, 'gpu_thread_type' : 'CUDA'}
        t.download_output_data = ['STDOUT', 'STDERR']

        # Add task to stage
        specfem_stage.add_tasks(t)
        
    p.add_stages(specfem_stage)
        
    res_dict = {
        'resource': 'princeton.traverse', # 'local.localhost',
        'schema'   : 'local',
        'walltime':  20, #2 * 30,
        'cpus': 16, #2 * 10 * 1,
        'gpus': 8, #2 * 4 * 2,          
    }

    appman = AppManager(hostname=hostname, port=port, username=username, password=password, resubmit_failed=False)
    appman.resource_desc = res_dict
    appman.workflow = set([p])
    appman.run()        
    

Tarball:

sandbox.tar.gz