Performance issue for large datasets

[maarten-ic]

When running IMAS-Validator on large datasets, we found that a lot of time is spent in finding all nodes with a value (link to source).

The main issue is that IMAS-Validator uses the (private) _path property of IDS nodes. This is slow when done on a large dataset: it scales quadratically in number of items in Arrays of Structures (see IMAS-Python source).

Reproduction

python -m cProfile -o profile.out -m imas_validator validate imas:hdf5?path=/work/imas/shared/TEST/simulations/test/8743dad6d7f211ef8fd59440c9e7706c/imasdb/iter/3/53298/2 --verbose --filter_ids edge_profiles

I have aborted this (with Ctrl+C) after ~55 minutes. The majority of that time was spent:

• 22 minutes in the Access Layer lowlevel to read the edge_profiles IDS. read_data() was called 38 million times
  • Total get() duration was ~27 minutes
• 29 minutes to determine the _path of 2.5 million IDS nodes
  • Probably this was not finished yet, as there are ~19 million nodes in the IDS. This includes structures and arrays of structures though, while _path is only determined for the data nodes.

To fix

It should be possible to obtain all filled IDS node paths in $O(N)$ instead of $O(N^2)$.