_structured_to_spmatrix() mapping fails for distributed solver

[lisa-gm]

When calling the distributed selected inversion the _structured_to_spmatrix() mapping fails. This fails gst_medium as well coreg test cases. So just calling run.py for gst_medium with 2 processes per solver will cause the error.

[lisa-gm]

#92

[vincent-maillou]

I tried to reproduce the error, running on Daint@Alps the gst_medium:
DALIA perfconfig:

# Set DALIA environment variables
export ARRAY_MODULE=cupy
export MPI_CUDA_AWARE=0
export USE_NCCL=0
export MPICH_GPU_SUPPORT_ENABLED=0

Results using sequential selected inversion:

--- Results ---
Theta values:
 [-0.0330793   2.22693606  1.36894052  1.39130633]
Covariance of theta:
 [[ 0.00002292 -0.00024479  0.00010828 -0.00001318]
 [-0.00024479  0.00638117 -0.00229961  0.00031113]
 [ 0.00010828 -0.00229961  0.00088043 -0.00010618]
 [-0.00001318  0.00031113 -0.00010618  0.00003477]]
Mean of the fixed effects:
 [-0.98412542  2.99886309  0.50024936  2.00139902  1.00313475 -1.99707928]

--- Comparisons ---
Norm (theta - theta_ref):         3.2330e-01
Norm (x - x_ref):                 2.7827e-01

--- Finished ---

Results using distributed selected inversion:

--- Results ---
Theta values:
 [-0.0331069   2.22687102  1.36892111  1.39130628]
Covariance of theta:
 [[ 0.00064419  0.00156197  0.00042576 -0.00000718]
 [ 0.00156197  0.00403221  0.0009594  -0.00000919]
 [ 0.00042576  0.0009594   0.00031812 -0.0000039 ]
 [-0.00000718 -0.00000919 -0.0000039   0.00002017]]
Mean of the fixed effects:
 [-0.98412658  2.99886309  0.50024936  2.00139902  1.00313475 -1.99707928]

--- Comparisons ---
Norm (theta - theta_ref):         3.2337e-01
Norm (x - x_ref):                 2.7862e-01

--- Finished ---

The covariance of theta matrix is different in both cases. I'm going to investigate the error in the branch https://github.com/dalia-project/DALIA/tree/fix_distributed_sellinv

[lisa-gm]

The covariance matrix computation is unrelated to the selected inversion. If you want to check that the covariance matrix works fine for the distributed solver it should be tested from a branch that includes the mpi_issues fix which hasn't been merged to dev, so we can rule out that this issue is unrelated of the distributed solver. in your output above i don't see if the selected inversion gets called at all. so im not sure if you actually called the _structured_to_spmatrix() mapping. can you confirm for sure that it got called?

[vincent-maillou]

Indeed, thanks I overlooked it.

So marginal_variance_latent <- self.get_marginal_variances_latent_parameters() <- self._compute_covariance_latent_parameters() <- self.solver.selected_inversion()

So here is a run using:

# Set DALIA environment variables
export ARRAY_MODULE=cupy
export MPI_CUDA_AWARE=0
export USE_NCCL=0
export MPICH_GPU_SUPPORT_ENABLED=0

Here is the result of using non-distributed selected inversion: Norm (marg var latent - ref): 1.5611e-09
Here is the result using distributed selected inversion: Norm (marg var latent - ref): 3.7804e-09

Is the difference above the one you referred to in this issue?
On what branch did you encounter this fail, in what run configuration?