mpi tests fail with otf

[sthibaul]

When running the mpi tests with otf backend and openmpi, I am getting crashes:

$ mpirun --host localhost:6 -np 6 bin/testEvent_mpi 2
Rank : 3 
Rank : 0 
Rank : 1 
Rank : 2 
Rank : 4 
Rank : 5 
[begin:3143993] *** Process received signal ***
[begin:3143993] Signal: Segmentation fault (11)
[begin:3143993] Signal code: Address not mapped (1)
[begin:3143993] Failing at address: 0x38
[begin:3143995] *** Process received signal ***
[begin:3143995] Signal: Segmentation fault (11)
[begin:3143995] Signal code: Address not mapped (1)
[begin:3143995] Failing at address: 0x38
[begin:3144000] *** Process received signal ***
[begin:3144000] Signal: Segmentation fault (11)
[begin:3144000] Signal code: Address not mapped (1)
[begin:3144000] Failing at address: 0x38
[begin:3144003] *** Process received signal ***
[begin:3144003] Signal: Segmentation fault (11)
[begin:3144003] Signal code: Address not mapped (1)
[begin:3144003] Failing at address: 0x38
[begin:3143998] *** Process received signal ***
[begin:3143998] Signal: Segmentation fault (11)
[begin:3143998] Signal code: Address not mapped (1)
[begin:3143998] Failing at address: 0x38
[begin:3143998] [ 0] [begin:3144000] [ 0] [begin:3143993] [ 0] [begin:3143995] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x3c510)[0x7ff558d09510]
[begin:3143998] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x3c510)[0x7f63ab855510]
[begin:3144000] [ 1] /lib/x86_64-linux-gnu/libgtg.so.0(OTFSetState+0x35)[0x7f63abb49795]
[begin:3144000] [ 2] bin/testEvent_mpi(+0x1ffb)[0x56365be19ffb]
[begin:3144000] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x3c510)[0x7fb56d303510]
[begin:3143993] [ 1] /lib/x86_64-linux-gnu/libgtg.so.0(OTFSetState+0x35)[0x7fb56d5f7795]
[begin:3143993] [ 2] bin/testEvent_mpi(+0x1ffb)[0x55f1e53baffb]
[begin:3143993] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x3c510)[0x7fbad68f5510]
[begin:3143995] [ 1] /lib/x86_64-linux-gnu/libgtg.so.0(OTFSetState+0x35)[0x7fbad6be9795]
[begin:3143995] [ 2] bin/testEvent_mpi(+0x1ffb)[0x55ae0a93affb]
[begin:3143995] [ 3] /lib/x86_64-linux-gnu/libgtg.so.0(OTFSetState+0x35)[0x7ff558ffd795]
[begin:3143998] [ 2] bin/testEvent_mpi(+0x1ffb)[0x55eb45074ffb]
[begin:3143998] [ 3] [begin:3144003] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x3c510)[0x7f7995fe7510]
[begin:3144003] [ 1] /lib/x86_64-linux-gnu/libgtg.so.0(OTFSetState+0x35)[0x7f79962db795]
[begin:3144003] [ 2] bin/testEvent_mpi(+0x1ffb)[0x55a09b1aeffb]
[begin:3144003] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x276ca)[0x7f7995fd26ca]
[begin:3144003] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f7995fd2785]
[begin:3144003] [ 5] bin/testEvent_mpi(+0x11b1)[0x55a09b1ae1b1]
[begin:3144003] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x276ca)[0x7fb56d2ee6ca]
[begin:3143993] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7fb56d2ee785]
[begin:3143993] [ 5] bin/testEvent_mpi(+0x11b1)[0x55f1e53ba1b1]
[begin:3143993] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x276ca)[0x7fbad68e06ca]
[begin:3143995] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7fbad68e0785]
[begin:3143995] [ 5] bin/testEvent_mpi(+0x11b1)[0x55ae0a93a1b1]
[begin:3143995] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x276ca)[0x7f63ab8406ca]
[begin:3144000] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f63ab840785]
[begin:3144000] [ 5] bin/testEvent_mpi(+0x11b1)[0x56365be191b1]
[begin:3144000] *** End of error message ***
/lib/x86_64-linux-gnu/libc.so.6(+0x276ca)[0x7ff558cf46ca]
[begin:3143998] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7ff558cf4785]
[begin:3143998] [ 5] bin/testEvent_mpi(+0x11b1)[0x55eb450741b1]
[begin:3143998] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 0 on node begin exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Running with the paje backend works fine:

$ mpirun --host localhost:6 -np 6 bin/testEvent_mpi 1
Rank : 0 
Rank : 1 
Rank : 2 
Rank : 3 
Rank : 4 
Rank : 5 

[trahay]

The problem happens when calling setState (but most events have the same problem) with a container that was defined by another container. In this case, the container data structure does not exist, and the library crashes.

Fixing this problem is quite time consuming, and I don't know if anyone uses the MPI feature of GTG. We could fix the mpi test program so that container are defined by MPI ranks instead of by the main rank.