Skip to content

Add MDAL_LF_SkipStatistics load flag and approximate min/max API - #528

Open
nicogodet wants to merge 6 commits into
lutraconsulting:masterfrom
nicogodet:approx-stats
Open

nicogodet wants to merge 6 commits into
lutraconsulting:masterfrom
nicogodet:approx-stats

Conversation

@nicogodet

@nicogodet nicogodet commented May 11, 2026

Copy link
Copy Markdown
Contributor

Lets callers opt out of the eager per-dataset statistics scan during MDAL_LoadMesh, dramatically reducing load time for large multi-timestep meshes when an exact min/max is not required upfront.

The gain is substantial!
I tested a "small" SELAFIN result file. We can have bigger ones which freeze QGIS for several minutes...

Fixes #526

Tested on QGIS with improvements to use this new API. Mesh file is stored on USB3 drive stick. I don't have access to network shared drive for now. My hardware is pretty old (i7-4790, 8Gb RAM).

Python snippet
from qgis.core import QgsMeshLayer, QgsProject, QgsSettings
from time import perf_counter

MESH_PATH = "/media/nicogodet/Nouveau nom/v5.res"
SAMPLE_COUNT = 10

s = QgsSettings()
key_on = "mesh/approximate-min-max-on-load"
key_n  = "mesh/approximate-min-max-sample-count"

def load_and_inspect(label):
  t0 = perf_counter()
  layer = QgsMeshLayer(MESH_PATH, f"v5_{label}", "mdal")
  elapsed = perf_counter() - t0
  if not layer.isValid():
      print(f"[{label}] invalid layer")
      return None, elapsed
  QgsProject.instance().addMapLayer(layer)

  provider = layer.dataProvider()
  n_groups = layer.datasetGroupCount()
  print(f"\n[{label}] load: {elapsed:.3f} s   groups: {n_groups}")
  print(f"  {'#':>3}  {'name':<32}  {'kind':<7}  {'ts':>6}  {'min':>14}  {'max':>14}")
  for g in range(n_groups):
      meta = provider.datasetGroupMetadata(g)
      n_ts = provider.datasetCount(g)
      kind = "scalar" if meta.isScalar() else "vector"
      print(f"  {g:>3}  {meta.name()[:32]:<32}  {kind:<7}  {n_ts:>6}  {meta.minimum():>14.6g}  {meta.maximum():>14.6g}")
  return layer, elapsed

# --- exact (toggle off) ---
s.setValue(key_on, False)
s.setValue(key_n, SAMPLE_COUNT)
QgsProject.instance().removeAllMapLayers()
_, t_exact = load_and_inspect("exact")

# --- approx (toggle on) ---
s.setValue(key_on, True)
QgsProject.instance().removeAllMapLayers()
_, t_approx = load_and_inspect(f"approx-{SAMPLE_COUNT}")

# --- résumé ---
print(f"\n>>> exact:                 {t_exact:.3f} s")
print(f">>> approx ({SAMPLE_COUNT} timesteps):   {t_approx:.3f} s")
print(f">>> speedup: x{t_exact / t_approx:.1f}  ({t_exact - t_approx:+.3f} s saved)")

Results

[exact] load: 44.877 s   groups: 7
    #  name                              kind         ts             min             max
    0  vitesse       ms                  vector      507               0         5.56107
    1  hauteur d'eau   m                 scalar      507               0          26.833
    2  surface libre   m                 scalar      507           -8.88           68.45
    3  fond            m                 scalar      507           -8.88           68.45
    4  frottement                        scalar      507               6              50
    5  cote maximum    m                 scalar      507           -8.88           68.45
    6  vitesse maximum ms                scalar      507               0         21.0616

[approx-10] load: 2.296 s   groups: 7
    #  name                              kind         ts             min             max
    0  vitesse       ms                  vector      507               0           3.911
    1  hauteur d'eau   m                 scalar      507               0          26.833
    2  surface libre   m                 scalar      507           -8.88           68.45
    3  fond            m                 scalar      507           -8.88           68.45
    4  frottement                        scalar      507               6              50
    5  cote maximum    m                 scalar      507           -8.88           68.45
    6  vitesse maximum ms                scalar      507               0         21.0616

exact:                 44.877 s
approx (10 timesteps):   2.296 s
speedup: x19.5  (+42.581 s saved)

AI disclaimer

Claude code (mostly Opus 5 and reviewed by Fable 5.1) done most of the work. I wrote the specifications (look at what GDAL does for rasters, dig in QGIS code base to learn the load path of rasters and meshes then look at MDAL to implement a similar approximative stats as GDAL does), Claude did the work. I did not reviewed all the changes but I read a bit to correct some comments and reduce verbosity.

@nicogodet

Copy link
Copy Markdown
Contributor Author

@uclaros Is it something you want to look at?
If so, please take a look at linked QGIS issue which explain why

@nicogodet
nicogodet force-pushed the approx-stats branch 2 times, most recently from a466215 to d7a164c Compare September 15, 2026 05:46
MDAL_LoadMeshWithFlags and MDAL_M_LoadDatasetsWithFlags take a bitwise
combination of MDAL_LoadFlag and pass it down to the driver, which can
read it back with Driver::loadFlags. The only flag so far,
MDAL_LF_SkipStatistics, has no effect yet.
Drivers now call setStatisticsIfRequired() instead of computing and
storing the range of every dataset and group at load time. Statistics
record whether they were actually computed, so nothing yet fills in the
ranges left empty by the flag.
MDAL_D_minimumMaximum and MDAL_G_minimumMaximum now compute the range on
first access, cache it and release the values read on the way. A range
built from an incomplete read is reported as unknown instead of being
cached, and a group in edit mode keeps following the datasets it holds.
The new entry point estimates the range of a group from a sample of
evenly spaced datasets, always including the first and the last one, and
falls back to the exact range when the sample would cover the whole group
or holds no valid value. The approximate range is never cached.
Drivers read on demand, so a file that changes under an open mesh handle
makes them throw from whichever entry point reads next, and an exception
crossing the extern "C" boundary kills the host application. The reading
entry points now report the failure instead.
The example external driver now tracks which dataset buffers MDAL asked
it to fill and can be told to report a failed read. Two tests use those
hooks to check that MDAL never releases a buffer it never requested and
never caches a range built from an incomplete read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add method to retrieve approximate statistics

1 participant