Repository navigation
Description
What problem does this solve?
Analysis tools guard against oversized requests with fixed limits, and there are 41 limit constants in crates/analysis/src. Two kinds of limit do different jobs:
- Memory caps stop an allocation the process can't survive:
MAX_PIXELS(64 M) for new rasters, Buffer's 2 M-vertex budget (fix(analysis): refine buffer arcs to meet reference tolerance #1241), the viewshed's 25 M-cell window, the interpolators' quarter ofMAX_PIXELS, andSCRATCH_BUDGETfor threads (perf(analysis): IDW and kriging rows on several cores #1273).- A failed allocation in Rust aborts the process. It isn't a panic, so
catch_unwindinanalysis::runcan't turn it into an error. In the browser no panic can be caught at all.
- A failed allocation in Rust aborts the process. It isn't a panic, so
- Work caps stop a run from hanging, since nothing can cancel a tool once it has started:
MAX_VIEWSHED_WORK(400 M ray steps)MAX_IDW_WORK(400 M cell-point pairs)MAX_KRIGING_WORK(40 M)MAX_DENSITY_WORK(50 M)- topology's
MAX_WORK(50 M) - network's
MAX_TRIES(50 M)
Both kinds are fixed, the same on a phone browser and a 64 GB workstation. Only trend::MAX_VALUES differs between wasm and desktop. That causes three problems:
- Desktop users hit limits their machine could handle. A viewshed on a 0.5 m swissALTI3D tile stops at about 290 m radius (0.5 m cells make the ray work grow with the cube of the radius). Kriging's work cap keeps 1,000 points to about 23,000 cells.
- The work caps were tuned for one core. The
MAX_VIEWSHED_WORKcomment says "each about a second or two". Since perf(analysis): one helper spreads a tool's items over the cores; Select by Location buffers on it #1254, perf(analysis): IDW and kriging rows on several cores #1273 and perf(analysis): viewshed rows on several cores #1283, the same work runs 5–11× faster on desktop, so the caps are now much stricter than intended. - Fixed caps don't cover combinations. Two moderate jobs running at once, or one job on a machine already holding large layers, can still run out of memory. Each passes its own cap.
Tools whose cost follows their input have no cap at all, and that's fine as long as the readers bound the input: overlay, erase/update, dissolve, zonal statistics, count points. A slow union of a few huge polygons can still run for a long time with no way to stop it.
Proposed solution
Three steps, each useful on its own:
1. Fallible allocation for the large buffers
- Use
Vec::try_reserve(andtry_reserve_exact) for output rasters, vertex arrays and per-thread scratch. When a reservation fails, return an error such as "not enough memory for a 40,000 × 30,000 raster" instead of aborting. - Where it helps:
- Browser: a failed
memory.growcomes back as an error. - Windows: committed memory is enforced, so the check is reliable.
- Linux: it catches obviously impossible requests. Overcommit can still let a reservation succeed and fail later.
- Browser: a failed
- No new dependency and no guessing. It also covers the combinations in problem 3.
2. Desktop caps scaled to the machine
- Memory caps become a fraction of physical RAM (for example a quarter), clamped between today's values as the minimum and a fixed maximum.
- Work caps scale with the threads the tool will use (
parallel::threads_for_rows). They could also simply be raised once step 3 lands. - The browser keeps today's fixed caps.
- Open questions:
- How to read physical memory:
sysinfo(MIT; checkdeny.tomland the wasm build), or small per-platform code (/proc/meminfo,GlobalMemoryStatusEx,sysctl hw.memsize). - Determinism. The same script or model could pass on one machine and fail on another. So:
- the limits should be readable, for example in
gp.envor alimitscommand; - errors should name the limit and its value on this machine;
- tests should pass fixed limits in.
- the limits should be readable, for example in
- How to read physical memory:
- GDAL does something similar:
GDAL_CACHEMAXdefaults to a percentage of physical RAM.
3. Cancellation and progress
With cancellation, desktop work caps could rise a lot or become warnings. Memory caps stay, because an abort can't be cancelled.
How tools run today:
gp.runcallsrun_tool(crates/engine/src/cmd/gp.rs), which builds the inputs from the session, callsrustgis_analysis::runand applies the result, all in one synchronous call.- On desktop that call runs on the UI thread, so the window waits until the tool finishes, and a Cancel button couldn't be clicked.
analysis::runtakes&dyn Inputs, which borrows the session's layers.
The pattern exists for one tool already. The STAC trend tool (analysispane.rs, cube.rs) runs on its own thread with cancel: &AtomicBool and done: &AtomicUsize, a Cancel button and a progress count. It drops a result that lands after another project has opened, and it reports a panic on its thread or a thread that can't start. This step would make that general:
| Part | Work | Size |
|---|---|---|
| a. A cancel flag reaching the tools | Inputs::cancelled(), default false. parallel::for_each_row and try_map check it alongside their stop flag. Long serial loops in the capped tools check it between rows or features (about 10–20 places). A cancelled run returns an error. |
small, nothing visible yet |
| b. Progress | A "done" counter beside the flag. The parallel helpers count rows themselves; other tools count features. | small, with (a) |
| c. Tools off the UI thread on desktop | The inputs mustn't borrow the session: share the layers, for example Arc for datasets. Run on a thread and land the result later through the same apply path, so undo, history and messages are unchanged. Decide what happens when an input layer is edited or removed, or another project opens, during a run. |
large: the main cost, likely 2–4 PRs across the engine and the Geoprocessing pane |
| d. The UI | Cancel and a progress bar in the Geoprocessing pane, as the trend tool has. | small once (c) exists |
What stays as it is:
- Scripts, models, the AI Assistant and the CLI call commands synchronously and wait for each result. They need no background runs, and (a) still lets them stop on a timeout or Ctrl-C.
- The browser has no threads here (the trend tool runs inline there too). Cancelling there would need a Web Worker, which is a separate project, so the browser keeps fixed caps.
Order: (a) and (b) first, since they're useful to the CLI and scripts on their own. Then (c) and (d) together.
Rules kept
- No
unwrap/expect/panic!outside tests.hostile_params_never_panicstays green, and still returns promptly with fixed test limits. - Everything below
apps/builds forwasm32-unknown-unknown. The browser keeps fixed caps. - Errors and notices say what was refused and why, with numbers.
Related
#1252 (analysis on several cores; raising MAX_IDW_WORK and MAX_KRIGING_WORK on desktop is listed there), #1241 (Buffer's vertex budget), #1227 (the Buffer benchmark that budget refuses past about 15,500 points), #1111 (large data).
Activity
Mavrick-F commentedon Nov 2, 2026
I’m working on the Buffer regression reported in #1227: retain the finer arcs and per-feature/dissolve complexity limits, but use a separate retained-storage byte budget and bounded batches for ordinary Buffer and distance selectors. This overlaps the Buffer part of this proposal, so I’ll link the fix here too. Machine-scaled limits, general allocation changes, cancellation and progress remain outside that patch. I found no open implementation PR; please flag if you have already started this Buffer piece so we can avoid duplicating it.
pick2510 commentedon Nov 2, 2026
I'm taking step 3 (cancellation and progress), in the order the proposal gives:
Steps 1 (fallible allocation) and 2 (limits scaled to the machine) stay open. @Mavrick-F, this doesn't touch your Buffer budget work in #1305.
feat(analysis): a running tool can be cancelled and reports its progr…
feat(analysis): a running tool can be cancelled and reports its progr…
pick2510 commentedon Nov 2, 2026
Step 3 (cancellation and progress) is done:
Steps 1 (fallible allocation) and 2 (limits scaled to the machine) are still open.
feat(analysis): an output raster the machine can't hold is an error, …
feat(data): ASCII grids, Zarr stores and pictures the machine can't h…
feat(data): GeoTIFF and COG rasters the machine can't hold are errors…
feat(engine): raster commands the machine can't hold are errors, not …
feat(data): the memory limits follow the computer's memory on the des…
feat(analysis): Viewshed, IDW and Kriging take as many times the work…
pick2510 commentedon Nov 3, 2026
Status (updated 7 November): steps 1, 2 and 3 are done, except topology's
MAX_WORKbelow.Step 1, fallible allocation:
A raster the machine can't hold is an error, not a crash.
Step 2, caps scaled to the machine:
gp.envreports the limits, and the errors name them.Step 3, cancellation and progress: #1311 and #1317.
Step 2, the last fixed memory caps: #1390 (merged 3 November): Buffer's byte budget, Kernel Density's output cells and the threads' scratch budget.
Left:
MAX_WORK. This waits for fix(analysis): measure radius searches on the ground near a pole, and meet at the poles (#1033) #1349 (fix(analysis): keep polar geometry through the geometry tools (#1033) #1347 has merged), which reworks how the topology checks count against it.Staying fixed:
network::MAX_TRIESbelongs to Create Random Points, not routing. It caps the point-in-polygon tries made from one seeded random stream. Scaling it would make the same seed give different points on different machines, so it stays as it is.feat(analysis): Buffer's budget, Kernel Density's cells and the threa…
feat(ui): the Analysis pane's Zonal Statistics, Viewshed and Interpol…
feat(ui): the Classification Wizard and Deep Learning run in the back…
feat(ui): the Analysis pane's single-pass raster tools run in the bac…
feat(analysis): a run that can be cancelled takes no work cap (#1291 …
feat(ui): Export Features and Export Raster write their file in the b…
feat(ui): a Zarr layer following the time slider reads its step off t…
feat(ui): a Zarr layer following the time slider reads its step off t…
feat(ui): Add Data and Load All read their file off the frame (#1291 …
fix(engine): a cancelled topology check stops between rules; a path p…
test(analysis): drop the cancelled-run block that couldn't fail (open…
fix(analysis): a topology check past its limit looks at Cancel every …
fix(engine): a cancelled topology check is an error, never a short re…
fix(engine): topology rules whose layer is gone are said; a cancelled…
feat(engine): the Error Inspector's topology check runs in the backgr…
pick2510 commentedon Nov 10, 2026
Done: the last item, topology's
MAX_WORK, landed in #1727.network::MAX_TRIESstays fixed, as said above. A related follow-up, #1735, makes deep learning say what a capped training left out.