Conversation
Contributor
Author
|
@uclaros Is it something you want to look at? |
nicogodet
force-pushed
the
approx-stats
branch
2 times, most recently
from
September 15, 2026 05:46
a466215 to
d7a164c
Compare
MDAL_LoadMeshWithFlags and MDAL_M_LoadDatasetsWithFlags take a bitwise combination of MDAL_LoadFlag and pass it down to the driver, which can read it back with Driver::loadFlags. The only flag so far, MDAL_LF_SkipStatistics, has no effect yet.
Drivers now call setStatisticsIfRequired() instead of computing and storing the range of every dataset and group at load time. Statistics record whether they were actually computed, so nothing yet fills in the ranges left empty by the flag.
MDAL_D_minimumMaximum and MDAL_G_minimumMaximum now compute the range on first access, cache it and release the values read on the way. A range built from an incomplete read is reported as unknown instead of being cached, and a group in edit mode keeps following the datasets it holds.
The new entry point estimates the range of a group from a sample of evenly spaced datasets, always including the first and the last one, and falls back to the exact range when the sample would cover the whole group or holds no valid value. The approximate range is never cached.
Drivers read on demand, so a file that changes under an open mesh handle makes them throw from whichever entry point reads next, and an exception crossing the extern "C" boundary kills the host application. The reading entry points now report the failure instead.
The example external driver now tracks which dataset buffers MDAL asked it to fill and can be told to report a failed read. Two tests use those hooks to check that MDAL never releases a buffer it never requested and never caches a range built from an incomplete read.
nicogodet
force-pushed
the
approx-stats
branch
from
September 15, 2026 09:11
d7a164c to
f75eb69
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets callers opt out of the eager per-dataset statistics scan during MDAL_LoadMesh, dramatically reducing load time for large multi-timestep meshes when an exact min/max is not required upfront.
The gain is substantial!
I tested a "small" SELAFIN result file. We can have bigger ones which freeze QGIS for several minutes...
Fixes #526
Tested on QGIS with improvements to use this new API. Mesh file is stored on USB3 drive stick. I don't have access to network shared drive for now. My hardware is pretty old (i7-4790, 8Gb RAM).
Python snippet
Results
AI disclaimer
Claude code (mostly Opus 5 and reviewed by Fable 5.1) done most of the work. I wrote the specifications (look at what GDAL does for rasters, dig in QGIS code base to learn the load path of rasters and meshes then look at MDAL to implement a similar approximative stats as GDAL does), Claude did the work. I did not reviewed all the changes but I read a bit to correct some comments and reduce verbosity.