Repository navigation
Description
Buffer now makes its features' buffers on several cores (#1248, geodesic_buffers in crates/analysis/src/geodesic.rs). On 100,000 points that cut 2.1 s on one core to 0.32 s on 40 threads (#1227). Most other expensive tools have the same shape: independent work per feature, cell or row. This issue tracks moving them over, and moving the desktop app's long runs off the UI thread (#1291 step 3).
The list comes from a read of crates/analysis on main (7903741). The gains are estimates until each one is measured.
To do
Track: cores runs a tool on several cores; UI thread runs it on a thread of its own on desktop, with Cancel and a progress row; no threads is a faster algorithm instead.
Nothing is left to do: every item is done or staying as it is.
Done
| Item | Track | PR | Measured (40 threads unless noted) |
|---|---|---|---|
Shared helper parallel::try_map |
cores | #1254, #1273, #1283 | see below |
| Buffer, with #1241's vertex budget charged per item | cores | #1248, #1254 | 100,000 points: 2.1 s → 0.32 s |
| Select by Location | cores | #1254 | 30,000 points by 10,000 selectors at 2 km: 0.84 s → 0.20 s |
| Kriging | cores | #1273 | 1,000 points onto 230 × 100 cells: 0.28 s → 0.08 s |
| IDW | cores | #1273 | 10,000 points onto 230 × 100 cells: 6.1 s → 0.55 s (QGIS 6.9 s, GDAL gdal_grid 0.16 s) |
| Viewshed | cores | #1283 | 15 km on a 1200 × 1200 DEM of 30 m cells: 1.55 s → 0.28 s (GDAL gdal_viewshed 0.23 s, QGIS grass:r.viewshed 2.7 s) |
| Deep-learning classify | cores | #1365 | 2000 × 2000 pixels, 4 bands, a 32-32 model, 3 classes: 5.21 s → 0.57 s |
| Slope, aspect, hillshade (#1111 step 6) | cores | #1368 | hillshade of a 9667 × 11947 DEM: 12.62 s → 1.39 s |
| Near | cores | #1369 | 100,000 points against 20,000 polygons: 7.40 s → 0.80 s |
| Zonal Statistics | cores | #1377 | 20,000 small zones 0.40 s → 0.10 s; 2,000 zones of 300 × 300 cells 3.13 s → 0.60 s (6 threads) |
| Resample and band math | cores | #1380 | Resample onto EPSG:2056 8.95 s → 0.82 s; NDVI on 4 bands 0.56 s → 0.09 s |
| Georeferencer residuals | cores | #1381 | spline at 250 points 1.50 s → 0.16 s; 3rd-order polynomial at 1,000 points 0.70 s → 0.07 s |
| Project Raster (and the Georeferencer's warp) | cores | #1382 | 4000 × 4000 RGB, lon/lat to UTM: nearest 3.28 s → 1.52 s, mode 7.56 s → 2.01 s |
Kernel Density, with MAX_DENSITY_WORK scaled by threads |
cores | #1384 | 200,000 points (540 M cell visits, refused on one thread): 0.37 s |
| k-means assignment | cores | #1384 | 4000 × 4000, 4 bands, 10 classes: 1.06 s → 0.22 s |
| Geoprocessing pane tools | UI thread | #1311, #1317 | |
| Analysis pane: Zonal Statistics, Viewshed, Interpolate; Viewshed from the map's menu | UI thread | #1404 | |
| Classification Wizard (k-means), Deep Learning training and classification | UI thread | #1412 | |
| Single-pass raster tools: raster functions and chains, Suitability, indices, change detection, supervised classification, process commands | UI thread | #1416 | |
| No work cap for runs that can be cancelled (Viewshed, IDW, Kriging, Kernel Density, Deep Learning) | caps | #1418 | |
| Export Features and Export Raster: progress and Cancel in the status bar; a cancelled export writes nothing | UI thread | #1433 | |
| Export Map and the layout exports, drawn on a copy of the project | UI thread | #1460 | |
| Zarr "follow time slider": each step read off the frame, playback waits for it | UI thread | #1469 | |
| Add Data (files, rasters, sheets, GDAL formats) and Load All read on a thread of their own, landing in the order asked; desktop only | UI thread | #1479 | |
| ModelBuilder and the Models list: a model's tools worked out on a copy of the session, the run taking each result whose inputs are unchanged; desktop only | UI thread | #1487 | |
| Script pane: a script's tools, models and session-only lines worked out on a copy the same way; desktop only | UI thread | #1503 | |
| Dissolve: each feature's group found by its key's hash | no threads | #1507 | 100,000 polygons in 50,000 groups: 7.76 s → 0.22 s |
| Zonal means and class means: only the cells in each zone's window tested | no threads | #1508 | a 160,000-cell zone on 4000 × 4000 cells: 0.062 s → 0.007 s (lon/lat), 0.534 s → 0.042 s (Web Mercator) |
| Band statistics: counted on several cores, the sums on one thread in the band's order | cores | #1509 | one 4000 × 4000 band: 0.21–0.26 s → 0.06–0.07 s |
| Feature To Point, Minimum Bounding Geometry without grouping, Calculate Geometry | cores | #1514 | 100,000 features of 65 vertices: Feature To Point 2.71 s → 0.22 s, Minimum Bounding Geometry 1.00 s → 0.30 s, Calculate Geometry 6.24 s → 0.63 s (polygons) and 4.90 s → 0.48 s (lines) |
| IDW at a power of 2: squared distances, no square root or power per cell and point | follow-up | #1520 | 10,000 points onto 230 × 100 cells from every point: 6.02 s → 0.88 s on one thread, 0.65 s → 0.10 s on all; 12 nearest, one thread: 0.82 s → 0.73 s |
| Symbology pane's Unique Values list: up to 200 classes listed (as many as an imported style can bring), and how many more it leaves out said | follow-up | #1543 | |
| Intersect, Identity, Union, Symmetrical Difference; the overlay zones Erase, Update and Split share | cores | #1740 | 100,000 polygons against 20,000 (64-vertex circles), 16 threads: Intersect 2.36 s → 0.29 s, Identity 4.09 s → 0.58 s, Union 4.39 s → 0.66 s, Symmetrical Difference 4.48 s → 0.60 s |
| Erase and Update | cores | #1744 | the same data, 16 threads: Erase 6.39 s → 0.73 s, Update 6.25 s → 0.72 s |
| Minimum Bounding Geometry with Group: All: each feature's hull points found on several cores, then one hull of them | cores | #1734 | |
| Clip: the masks unioned, and each feature clipped, on several cores, with a cache of mask unions per thread | cores | #1779 | 100,000 circles (64 vertices) clipped by 20,000 overlapping ones, 16 threads: 11.1 s → about 1.0 s |
Split: the masks unioned (one area a thread), and each feature's part in each area, on several cores; parallel::try_map_each_on for a few long items |
cores | #1783 | 100,000 circles, 16 threads: split by 20 areas into 10 layers 3.83 s → 0.64 s, by 1,000 areas 3.78 s → 0.67 s, by 8 areas of 2,000 detailed circles each 16.90 s → 2.06 s |
| Spatial Join, Count Points / Summarize Within, Simplify | cores | #1746 | 100,000 polygons against 20,000 (64-vertex circles), 16 threads: Spatial Join Intersects 2.39 s → 0.44 s, Largest overlap 2.52 s → 0.51 s, Closest 99.12 s → 7.51 s; Summarize Within (1,000,000 points) 0.67 s → 0.29 s; Simplify 0.92 s → 0.16 s |
Topology check "not covered" (Must Be Covered By): each point looked up in an ExtentIndex of the polygons' boxes |
no threads | #1928 | 98,596 parcels, one thread: 10,000 points 1.51 s → 0.11 s, 100,000 points 14.48 s → 0.17 s; the same points reported |
Viewshed accuracy: heights read bilinearly between cells along each ray, as GRASS's r.viewshed does |
follow-up | #1789, #1835 |
Staying as they are
| Item | Why |
|---|---|
| Shortest route (Dijkstra), network training | Sequential. |
| Topology checks: overlaps, self-intersections, dangles, gaps | Already quick: on 98,429 parcels of 17 vertices and 49,928 road segments, overlaps take 0.2 s, self-intersections 0.15 s, dangles 0.07 s and gaps 0.5 s (one union). Their work and error limits stop the scan at a point that depends on order, which threads would have to keep. The desktop app checks them on a thread of its own with Cancel (#1291). |
| Random points | Come from one seeded random stream. |
| The A5 grid, lasso selection | Already capped small. |
| Merge, the attribute joins, the table statistics | Limited by hashing or memory. |
remap, weighted_overlay, classify_min_distance on cores |
Limited by memory. |
| One big union at the end (Buffer with Dissolve All, gaps) | A tree of unions could change the output vertices unless the partition is fixed. Large; maybe not worth it. |
| Edit Pixels off the UI thread | An edit made by hand, so it should stay immediate. |
| Georeference from the pane off the UI thread | The pane does no warp, so there's nothing to move. |
Rules for the cores track
Each conversion keeps what Buffer's does:
- Same output: results stay in input order and byte for byte the same as one thread. Check this against
mainthe way perf(analysis): geodesic buffers take each vertex back once and look at fewer edges, on all cores #1248 did. - Browser: one thread there (
cfg!(target_arch = "wasm32")). - Thresholds: only above a size where threads pay off. At most 16 threads, and the calling thread works too.
- Errors and panics: the first error in input order is the one reported. A worker's panic is raised again for
analysis::runto catch. Notices still give one count per kind. - No new dependency: use
std::thread::scope.
The shared helper (#1254)
parallel::try_map(items, min_items, scratch, f) in crates/analysis/src/parallel.rs. f gets each item and its index, and returns Result<R, E>. The helper works like this:
- Threads take the next 16 items from a shared counter, the calling thread among them, so expensive items lying together are spread out.
- Results are placed by index.
- Each thread makes one scratch value (the kriging and IDW buffers, count vectors).
- For rasters, the items are row numbers (
parallel::for_each_row). - The first error stops the threads after the items they are making.
Closures must capture the &Dataset, &Raster and parameters, not &dyn Inputs, which isn't Sync.
#1254's two leftovers (a drop guard so a worker's panic stops the others, and a test for Select by Location's selecting-layer notice) landed in #1273. #1273's two leftovers (try_map_on back to get(), and for_each_row taking a few rows per lock, shrinking at the end) landed in #1283.
Related
- Analysis limits: fail gracefully on allocation, scale with the machine, and cancel long runs #1291: large data (fallible allocation, caps and cancellation). Its step 3 is the UI-thread track here.
- Tracking: performance gaps from Geoawesome's QGIS benchmark (buffer, raster, drawing) #1227: Buffer and the article's gaps.
- Tracking: large data (layers over 2 million features, slow hillshade and GeoTIFF writing) #1111: large data; step 6 is threads for raster kernels and COG tiles.
- [Feature]: Draw a heavy layer on several CPU cores #1067: drawing on several cores, which is outside
analysis.
Activity
pick2510 commentedon Nov 1, 2026
I'll take step 1: the shared helper (generalising
buffers_on), with Select by Location's buffers as its first user. Then I'll move Buffer onto it too, so there's one copy of the threading code.perf(analysis): one helper spreads a tool's items over the cores; Sel…
perf(analysis): one helper spreads a tool's items over the cores; Sel…
pick2510 commentedon Nov 1, 2026
I'll take the first two items of step 2, Kriging and IDW. Both work out each cell on its own, so they share one change: rows on
parallel::try_map, with a scratchKrigingBuffersper thread for kriging. The same change carries #1254's two leftovers: a drop guard so a worker's panic stops the others, and a test for Select by Location's selecting-layer notice. I'll check the output byte for byte againstmainand time IDW against QGIS 4.2.3 (qgis:idwinterpolation, GDAL'sgdal_grid) on synthetic points. QGIS has no kriging tool without SAGA, so kriging is timed only againstmain. RaisingMAX_KRIGING_WORKandMAX_IDW_WORKcomes after, once there are numbers.pick2510 commentedon Nov 1, 2026
I'll take Viewshed next (step 2's third item). Each cell within the radius walks its own ray and writes only its own cell, so its rows go on
parallel::for_each_rowas IDW and kriging do since #1273, with the output the same bit for bit. The same change carries #1273's two leftovers:try_map_onback toget(), andfor_each_rowtaking a few rows per lock on tall, narrow grids. I'll time it on a synthetic DEM againstmain, GDAL'sgdal_viewshedand QGIS (gdal:viewshed,grass:r.viewshed). GDAL and GRASS sweep outward with a horizon per cell instead of walking a ray per cell, so if the gap stays large after threads, a sweep algorithm would be a change of its own.perf(analysis): deep-learning classification rows on several cores (#…
perf(analysis): slope, aspect and hillshade rows on several cores (#1252
perf(analysis): Near searches its features on several cores (#1252) (#…
perf(analysis): Zonal Statistics works out its zones on several cores (…
perf(analysis): Resample and band math make their rows on several cor…
perf(analysis): the Georeferencer's leave-one-out refits run on sever…
perf(analysis): Project Raster and the georeferencer's warp make thei…
perf(analysis): k-means and Kernel Density make their rows on several…
perf(analysis): Intersect, Identity, Union and Symmetrical Difference…
perf(analysis): Erase and Update work on several cores (#1252) (#1744)
fix(analysis): bound Spatial Join's held matches per thread; Summariz…
perf(analysis): Spatial Join, Summarize Within and Simplify work on s…
perf(analysis): Clip on several cores (#1252) (#1779)
perf(analysis): Split on several cores (#1252) (#1783)
perf(analysis): Must Be Covered By looks each point up in an index of…
pick2510 commentedon Nov 13, 2026
Done: the last two items landed in #1928 (Must Be Covered By with a spatial index) and #1789/#1835 (Viewshed's bilinear heights). Every other item is done or staying as it is, with the reasons in the tables above.