Skip to content

Analysis limits: fail gracefully on allocation, scale with the machine, and cancel long runs #1291

Closed

Description

@pick2510

What problem does this solve?

Analysis tools guard against oversized requests with fixed limits, and there are 41 limit constants in crates/analysis/src. Two kinds of limit do different jobs:

  • Memory caps stop an allocation the process can't survive: MAX_PIXELS (64 M) for new rasters, Buffer's 2 M-vertex budget (fix(analysis): refine buffer arcs to meet reference tolerance #1241), the viewshed's 25 M-cell window, the interpolators' quarter of MAX_PIXELS, and SCRATCH_BUDGET for threads (perf(analysis): IDW and kriging rows on several cores #1273).
    • A failed allocation in Rust aborts the process. It isn't a panic, so catch_unwind in analysis::run can't turn it into an error. In the browser no panic can be caught at all.
  • Work caps stop a run from hanging, since nothing can cancel a tool once it has started:
    • MAX_VIEWSHED_WORK (400 M ray steps)
    • MAX_IDW_WORK (400 M cell-point pairs)
    • MAX_KRIGING_WORK (40 M)
    • MAX_DENSITY_WORK (50 M)
    • topology's MAX_WORK (50 M)
    • network's MAX_TRIES (50 M)

Both kinds are fixed, the same on a phone browser and a 64 GB workstation. Only trend::MAX_VALUES differs between wasm and desktop. That causes three problems:

  1. Desktop users hit limits their machine could handle. A viewshed on a 0.5 m swissALTI3D tile stops at about 290 m radius (0.5 m cells make the ray work grow with the cube of the radius). Kriging's work cap keeps 1,000 points to about 23,000 cells.
  2. The work caps were tuned for one core. The MAX_VIEWSHED_WORK comment says "each about a second or two". Since perf(analysis): one helper spreads a tool's items over the cores; Select by Location buffers on it #1254, perf(analysis): IDW and kriging rows on several cores #1273 and perf(analysis): viewshed rows on several cores #1283, the same work runs 5–11× faster on desktop, so the caps are now much stricter than intended.
  3. Fixed caps don't cover combinations. Two moderate jobs running at once, or one job on a machine already holding large layers, can still run out of memory. Each passes its own cap.

Tools whose cost follows their input have no cap at all, and that's fine as long as the readers bound the input: overlay, erase/update, dissolve, zonal statistics, count points. A slow union of a few huge polygons can still run for a long time with no way to stop it.

Proposed solution

Three steps, each useful on its own:

1. Fallible allocation for the large buffers

  • Use Vec::try_reserve (and try_reserve_exact) for output rasters, vertex arrays and per-thread scratch. When a reservation fails, return an error such as "not enough memory for a 40,000 × 30,000 raster" instead of aborting.
  • Where it helps:
    • Browser: a failed memory.grow comes back as an error.
    • Windows: committed memory is enforced, so the check is reliable.
    • Linux: it catches obviously impossible requests. Overcommit can still let a reservation succeed and fail later.
  • No new dependency and no guessing. It also covers the combinations in problem 3.

2. Desktop caps scaled to the machine

  • Memory caps become a fraction of physical RAM (for example a quarter), clamped between today's values as the minimum and a fixed maximum.
  • Work caps scale with the threads the tool will use (parallel::threads_for_rows). They could also simply be raised once step 3 lands.
  • The browser keeps today's fixed caps.
  • Open questions:
    • How to read physical memory: sysinfo (MIT; check deny.toml and the wasm build), or small per-platform code (/proc/meminfo, GlobalMemoryStatusEx, sysctl hw.memsize).
    • Determinism. The same script or model could pass on one machine and fail on another. So:
      • the limits should be readable, for example in gp.env or a limits command;
      • errors should name the limit and its value on this machine;
      • tests should pass fixed limits in.
  • GDAL does something similar: GDAL_CACHEMAX defaults to a percentage of physical RAM.

3. Cancellation and progress

With cancellation, desktop work caps could rise a lot or become warnings. Memory caps stay, because an abort can't be cancelled.

How tools run today:

  • gp.run calls run_tool (crates/engine/src/cmd/gp.rs), which builds the inputs from the session, calls rustgis_analysis::run and applies the result, all in one synchronous call.
  • On desktop that call runs on the UI thread, so the window waits until the tool finishes, and a Cancel button couldn't be clicked.
  • analysis::run takes &dyn Inputs, which borrows the session's layers.

The pattern exists for one tool already. The STAC trend tool (analysispane.rs, cube.rs) runs on its own thread with cancel: &AtomicBool and done: &AtomicUsize, a Cancel button and a progress count. It drops a result that lands after another project has opened, and it reports a panic on its thread or a thread that can't start. This step would make that general:

Part Work Size
a. A cancel flag reaching the tools Inputs::cancelled(), default false. parallel::for_each_row and try_map check it alongside their stop flag. Long serial loops in the capped tools check it between rows or features (about 10–20 places). A cancelled run returns an error. small, nothing visible yet
b. Progress A "done" counter beside the flag. The parallel helpers count rows themselves; other tools count features. small, with (a)
c. Tools off the UI thread on desktop The inputs mustn't borrow the session: share the layers, for example Arc for datasets. Run on a thread and land the result later through the same apply path, so undo, history and messages are unchanged. Decide what happens when an input layer is edited or removed, or another project opens, during a run. large: the main cost, likely 2–4 PRs across the engine and the Geoprocessing pane
d. The UI Cancel and a progress bar in the Geoprocessing pane, as the trend tool has. small once (c) exists

What stays as it is:

  • Scripts, models, the AI Assistant and the CLI call commands synchronously and wait for each result. They need no background runs, and (a) still lets them stop on a timeout or Ctrl-C.
  • The browser has no threads here (the trend tool runs inline there too). Cancelling there would need a Web Worker, which is a separate project, so the browser keeps fixed caps.

Order: (a) and (b) first, since they're useful to the CLI and scripts on their own. Then (c) and (d) together.

Rules kept

  • No unwrap/expect/panic! outside tests. hostile_params_never_panic stays green, and still returns promptly with fixed test limits.
  • Everything below apps/ builds for wasm32-unknown-unknown. The browser keeps fixed caps.
  • Errors and notices say what was refused and why, with numbers.

Related

#1252 (analysis on several cores; raising MAX_IDW_WORK and MAX_KRIGING_WORK on desktop is listed there), #1241 (Buffer's vertex budget), #1227 (the Buffer benchmark that budget refuses past about 15,500 points), #1111 (large data).

Activity

  1. Mavrick-F commented on Nov 2, 2026

    @Mavrick-F
    Member

    I’m working on the Buffer regression reported in #1227: retain the finer arcs and per-feature/dissolve complexity limits, but use a separate retained-storage byte budget and bounded batches for ordinary Buffer and distance selectors. This overlaps the Buffer part of this proposal, so I’ll link the fix here too. Machine-scaled limits, general allocation changes, cancellation and progress remain outside that patch. I found no open implementation PR; please flag if you have already started this Buffer piece so we can avoid duplicating it.

  2. pick2510 commented on Nov 2, 2026

    @pick2510
    ContributorAuthor

    I'm taking step 3 (cancellation and progress), in the order the proposal gives:

    1. A cancel flag and progress counter reaching the tools (3a and 3b). These are useful to the CLI and scripts on their own.
    2. Tools running off the UI thread on desktop (3c), with Cancel and a progress bar in the Geoprocessing pane (3d). This will probably take a few PRs.

    Steps 1 (fallible allocation) and 2 (limits scaled to the machine) stay open. @Mavrick-F, this doesn't touch your Buffer budget work in #1305.

  3. pick2510 commented on Nov 2, 2026

    @pick2510
    ContributorAuthor

    Step 3 (cancellation and progress) is done:

    Steps 1 (fallible allocation) and 2 (limits scaled to the machine) are still open.

  4. pick2510 commented on Nov 3, 2026

    @pick2510
    ContributorAuthor

    Status (updated 7 November): steps 1, 2 and 3 are done, except topology's MAX_WORK below.

    Step 1, fallible allocation:

    A raster the machine can't hold is an error, not a crash.

    Step 2, caps scaled to the machine:

    Step 3, cancellation and progress: #1311 and #1317.

    Step 2, the last fixed memory caps: #1390 (merged 3 November): Buffer's byte budget, Kernel Density's output cells and the threads' scratch budget.

    Left:

    Staying fixed: network::MAX_TRIES belongs to Create Random Points, not routing. It caps the point-in-polygon tries made from one seeded random stream. Scaling it would make the same seed give different points on different machines, so it stays as it is.

  5. pick2510 commented on Nov 10, 2026

    @pick2510
    ContributorAuthor

    Done: the last item, topology's MAX_WORK, landed in #1727.

    network::MAX_TRIES stays fixed, as said above. A related follow-up, #1735, makes deep learning say what a capped training left out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions