Skip to content

Tracking: analysis tools on several cores #1252

Closed

Description

@pick2510

Buffer now makes its features' buffers on several cores (#1248, geodesic_buffers in crates/analysis/src/geodesic.rs). On 100,000 points that cut 2.1 s on one core to 0.32 s on 40 threads (#1227). Most other expensive tools have the same shape: independent work per feature, cell or row. This issue tracks moving them over, and moving the desktop app's long runs off the UI thread (#1291 step 3).

The list comes from a read of crates/analysis on main (7903741). The gains are estimates until each one is measured.

To do

Track: cores runs a tool on several cores; UI thread runs it on a thread of its own on desktop, with Cancel and a progress row; no threads is a faster algorithm instead.

Nothing is left to do: every item is done or staying as it is.

Done

Item Track PR Measured (40 threads unless noted)
Shared helper parallel::try_map cores #1254, #1273, #1283 see below
Buffer, with #1241's vertex budget charged per item cores #1248, #1254 100,000 points: 2.1 s → 0.32 s
Select by Location cores #1254 30,000 points by 10,000 selectors at 2 km: 0.84 s → 0.20 s
Kriging cores #1273 1,000 points onto 230 × 100 cells: 0.28 s → 0.08 s
IDW cores #1273 10,000 points onto 230 × 100 cells: 6.1 s → 0.55 s (QGIS 6.9 s, GDAL gdal_grid 0.16 s)
Viewshed cores #1283 15 km on a 1200 × 1200 DEM of 30 m cells: 1.55 s → 0.28 s (GDAL gdal_viewshed 0.23 s, QGIS grass:r.viewshed 2.7 s)
Deep-learning classify cores #1365 2000 × 2000 pixels, 4 bands, a 32-32 model, 3 classes: 5.21 s → 0.57 s
Slope, aspect, hillshade (#1111 step 6) cores #1368 hillshade of a 9667 × 11947 DEM: 12.62 s → 1.39 s
Near cores #1369 100,000 points against 20,000 polygons: 7.40 s → 0.80 s
Zonal Statistics cores #1377 20,000 small zones 0.40 s → 0.10 s; 2,000 zones of 300 × 300 cells 3.13 s → 0.60 s (6 threads)
Resample and band math cores #1380 Resample onto EPSG:2056 8.95 s → 0.82 s; NDVI on 4 bands 0.56 s → 0.09 s
Georeferencer residuals cores #1381 spline at 250 points 1.50 s → 0.16 s; 3rd-order polynomial at 1,000 points 0.70 s → 0.07 s
Project Raster (and the Georeferencer's warp) cores #1382 4000 × 4000 RGB, lon/lat to UTM: nearest 3.28 s → 1.52 s, mode 7.56 s → 2.01 s
Kernel Density, with MAX_DENSITY_WORK scaled by threads cores #1384 200,000 points (540 M cell visits, refused on one thread): 0.37 s
k-means assignment cores #1384 4000 × 4000, 4 bands, 10 classes: 1.06 s → 0.22 s
Geoprocessing pane tools UI thread #1311, #1317
Analysis pane: Zonal Statistics, Viewshed, Interpolate; Viewshed from the map's menu UI thread #1404
Classification Wizard (k-means), Deep Learning training and classification UI thread #1412
Single-pass raster tools: raster functions and chains, Suitability, indices, change detection, supervised classification, process commands UI thread #1416
No work cap for runs that can be cancelled (Viewshed, IDW, Kriging, Kernel Density, Deep Learning) caps #1418
Export Features and Export Raster: progress and Cancel in the status bar; a cancelled export writes nothing UI thread #1433
Export Map and the layout exports, drawn on a copy of the project UI thread #1460
Zarr "follow time slider": each step read off the frame, playback waits for it UI thread #1469
Add Data (files, rasters, sheets, GDAL formats) and Load All read on a thread of their own, landing in the order asked; desktop only UI thread #1479
ModelBuilder and the Models list: a model's tools worked out on a copy of the session, the run taking each result whose inputs are unchanged; desktop only UI thread #1487
Script pane: a script's tools, models and session-only lines worked out on a copy the same way; desktop only UI thread #1503
Dissolve: each feature's group found by its key's hash no threads #1507 100,000 polygons in 50,000 groups: 7.76 s → 0.22 s
Zonal means and class means: only the cells in each zone's window tested no threads #1508 a 160,000-cell zone on 4000 × 4000 cells: 0.062 s → 0.007 s (lon/lat), 0.534 s → 0.042 s (Web Mercator)
Band statistics: counted on several cores, the sums on one thread in the band's order cores #1509 one 4000 × 4000 band: 0.21–0.26 s → 0.06–0.07 s
Feature To Point, Minimum Bounding Geometry without grouping, Calculate Geometry cores #1514 100,000 features of 65 vertices: Feature To Point 2.71 s → 0.22 s, Minimum Bounding Geometry 1.00 s → 0.30 s, Calculate Geometry 6.24 s → 0.63 s (polygons) and 4.90 s → 0.48 s (lines)
IDW at a power of 2: squared distances, no square root or power per cell and point follow-up #1520 10,000 points onto 230 × 100 cells from every point: 6.02 s → 0.88 s on one thread, 0.65 s → 0.10 s on all; 12 nearest, one thread: 0.82 s → 0.73 s
Symbology pane's Unique Values list: up to 200 classes listed (as many as an imported style can bring), and how many more it leaves out said follow-up #1543
Intersect, Identity, Union, Symmetrical Difference; the overlay zones Erase, Update and Split share cores #1740 100,000 polygons against 20,000 (64-vertex circles), 16 threads: Intersect 2.36 s → 0.29 s, Identity 4.09 s → 0.58 s, Union 4.39 s → 0.66 s, Symmetrical Difference 4.48 s → 0.60 s
Erase and Update cores #1744 the same data, 16 threads: Erase 6.39 s → 0.73 s, Update 6.25 s → 0.72 s
Minimum Bounding Geometry with Group: All: each feature's hull points found on several cores, then one hull of them cores #1734
Clip: the masks unioned, and each feature clipped, on several cores, with a cache of mask unions per thread cores #1779 100,000 circles (64 vertices) clipped by 20,000 overlapping ones, 16 threads: 11.1 s → about 1.0 s
Split: the masks unioned (one area a thread), and each feature's part in each area, on several cores; parallel::try_map_each_on for a few long items cores #1783 100,000 circles, 16 threads: split by 20 areas into 10 layers 3.83 s → 0.64 s, by 1,000 areas 3.78 s → 0.67 s, by 8 areas of 2,000 detailed circles each 16.90 s → 2.06 s
Spatial Join, Count Points / Summarize Within, Simplify cores #1746 100,000 polygons against 20,000 (64-vertex circles), 16 threads: Spatial Join Intersects 2.39 s → 0.44 s, Largest overlap 2.52 s → 0.51 s, Closest 99.12 s → 7.51 s; Summarize Within (1,000,000 points) 0.67 s → 0.29 s; Simplify 0.92 s → 0.16 s
Topology check "not covered" (Must Be Covered By): each point looked up in an ExtentIndex of the polygons' boxes no threads #1928 98,596 parcels, one thread: 10,000 points 1.51 s → 0.11 s, 100,000 points 14.48 s → 0.17 s; the same points reported
Viewshed accuracy: heights read bilinearly between cells along each ray, as GRASS's r.viewshed does follow-up #1789, #1835

Staying as they are

Item Why
Shortest route (Dijkstra), network training Sequential.
Topology checks: overlaps, self-intersections, dangles, gaps Already quick: on 98,429 parcels of 17 vertices and 49,928 road segments, overlaps take 0.2 s, self-intersections 0.15 s, dangles 0.07 s and gaps 0.5 s (one union). Their work and error limits stop the scan at a point that depends on order, which threads would have to keep. The desktop app checks them on a thread of its own with Cancel (#1291).
Random points Come from one seeded random stream.
The A5 grid, lasso selection Already capped small.
Merge, the attribute joins, the table statistics Limited by hashing or memory.
remap, weighted_overlay, classify_min_distance on cores Limited by memory.
One big union at the end (Buffer with Dissolve All, gaps) A tree of unions could change the output vertices unless the partition is fixed. Large; maybe not worth it.
Edit Pixels off the UI thread An edit made by hand, so it should stay immediate.
Georeference from the pane off the UI thread The pane does no warp, so there's nothing to move.

Rules for the cores track

Each conversion keeps what Buffer's does:

  • Same output: results stay in input order and byte for byte the same as one thread. Check this against main the way perf(analysis): geodesic buffers take each vertex back once and look at fewer edges, on all cores #1248 did.
  • Browser: one thread there (cfg!(target_arch = "wasm32")).
  • Thresholds: only above a size where threads pay off. At most 16 threads, and the calling thread works too.
  • Errors and panics: the first error in input order is the one reported. A worker's panic is raised again for analysis::run to catch. Notices still give one count per kind.
  • No new dependency: use std::thread::scope.
The shared helper (#1254)

parallel::try_map(items, min_items, scratch, f) in crates/analysis/src/parallel.rs. f gets each item and its index, and returns Result<R, E>. The helper works like this:

  • Threads take the next 16 items from a shared counter, the calling thread among them, so expensive items lying together are spread out.
  • Results are placed by index.
  • Each thread makes one scratch value (the kriging and IDW buffers, count vectors).
  • For rasters, the items are row numbers (parallel::for_each_row).
  • The first error stops the threads after the items they are making.

Closures must capture the &Dataset, &Raster and parameters, not &dyn Inputs, which isn't Sync.

#1254's two leftovers (a drop guard so a worker's panic stops the others, and a test for Select by Location's selecting-layer notice) landed in #1273. #1273's two leftovers (try_map_on back to get(), and for_each_row taking a few rows per lock, shrinking at the end) landed in #1283.

Related

Activity

  1. pinned this issue on Nov 1, 2026
  2. pick2510 commented on Nov 1, 2026

    @pick2510
    ContributorAuthor

    I'll take step 1: the shared helper (generalising buffers_on), with Select by Location's buffers as its first user. Then I'll move Buffer onto it too, so there's one copy of the threading code.

  3. self-assigned this
    on Nov 1, 2026
  4. pick2510 commented on Nov 1, 2026

    @pick2510
    ContributorAuthor

    I'll take the first two items of step 2, Kriging and IDW. Both work out each cell on its own, so they share one change: rows on parallel::try_map, with a scratch KrigingBuffers per thread for kriging. The same change carries #1254's two leftovers: a drop guard so a worker's panic stops the others, and a test for Select by Location's selecting-layer notice. I'll check the output byte for byte against main and time IDW against QGIS 4.2.3 (qgis:idwinterpolation, GDAL's gdal_grid) on synthetic points. QGIS has no kriging tool without SAGA, so kriging is timed only against main. Raising MAX_KRIGING_WORK and MAX_IDW_WORK comes after, once there are numbers.

  5. pick2510 commented on Nov 1, 2026

    @pick2510
    ContributorAuthor

    I'll take Viewshed next (step 2's third item). Each cell within the radius walks its own ray and writes only its own cell, so its rows go on parallel::for_each_row as IDW and kriging do since #1273, with the output the same bit for bit. The same change carries #1273's two leftovers: try_map_on back to get(), and for_each_row taking a few rows per lock on tall, narrow grids. I'll time it on a synthetic DEM against main, GDAL's gdal_viewshed and QGIS (gdal:viewshed, grass:r.viewshed). GDAL and GRASS sweep outward with a horizon per cell instead of walking a ray per cell, so if the gap stays large after threads, a sweep algorithm would be a change of its own.

  6. unpinned this issue on Nov 12, 2026
  7. pick2510 commented on Nov 13, 2026

    @pick2510
    ContributorAuthor

    Done: the last two items landed in #1928 (Must Be Covered By with a spatial index) and #1789/#1835 (Viewshed's bilinear heights). Every other item is done or staying as it is, with the reasons in the tables above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions