Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,30 @@ Before picking work, read [ROADMAP.md](ROADMAP.md) (check Underway) and
what this version couldn't read; a reader checks the version it is given
([saved files](docs/maintaining.md#saved-files-and-settings)).

## Never hold the window
On the desktop, a command whose work grows with its input (features, cells, tiles, files,
requests) runs off the UI thread, with progress and Cancel (#1562). The browser has one thread and
is out of scope.
- Use an existing background path: tools go through `gp::tool_start`; exports through
`exports::runs_in_background`; commands that make a raster through
`workflows::runs_in_background`; models and scripts through the copy (`workflows::ahead_on_copy`),
with the command on `ON_A_COPY` when running it twice changes nothing outside the session.
- The work looks at Cancel between items of its long loops (`control::check`, `Pace`,
`progress::check`) and counts its progress. Where progress can't be measured, a busy indicator
without a count is fine; Cancel is not optional.
- Work that splits into independent parts runs on several threads where practical: the same
output as one thread, bit for bit; one thread below a size threshold and in the browser; the
shared helpers (`parallel::for_each_row`, `try_map`, `try_map_on`); the thread count from
`rustgis_data::threads` (it follows the user's setting), never `available_parallelism` or a
fixed number; scratch memory within `memory::thread_scratch`; a thread the OS refuses costs
speed, not the run (`Builder::spawn_scoped`).
- A command that must run on the UI thread says why in its doc comment.
- `crates/engine/src/tests.rs::long_commands_stop_when_cancelled_while_they_run` cancels each
background command as its work starts and fails one that never looks; its lists must match the
background registries, so a new background command is added there. Passing shows the work looks
at Cancel, not that it looks often: a long step without a check (one union, one read) is said in
its doc comment.

## Architecture
| Layer | Crate | Job |
|---|---|---|
Expand Down
2 changes: 2 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ something on the roadmap (or not on it) interests you, propose it and open a pul
- what you plan to build: the commands it adds, where it sits in the ribbon (following the
[ribbon guidelines](docs/contributing.md#ribbon-guidelines)), which formats or tools are involved;
- which crates it touches and whether it adds a dependency;
- for work that grows with the data, how it runs off the UI thread with progress and Cancel
(see [Never hold the window](AGENTS.md#never-hold-the-window));
- how you will test it.

Put it in an issue with the
Expand Down
61 changes: 58 additions & 3 deletions crates/analysis/src/control.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ pub struct Control {
total: AtomicU64,
/// Someone can stop the run while it goes ([`Control::stoppable`]).
stoppable: bool,
/// Cancelled as soon as its run starts ([`Control::cancelled_once_started`]).
cancel_at_start: bool,
/// How often the work looked at the flag ([`Control::looked`]).
looked: AtomicU64,
}

impl Control {
Expand All @@ -34,6 +38,37 @@ impl Control {
Arc::new(Control { stoppable: !cfg!(target_arch = "wasm32"), ..Control::default() })
}

/// A control that [`crate::run_under`] cancels as soon as the work has started, past the
/// check before it: the work's own looks at the flag ([`Control::looked`]) decide whether it
/// stops. For tests that a long command checks for Cancel (#1562), with no timing.
pub fn cancelled_once_started() -> Arc<Control> {
Arc::new(Control { cancel_at_start: true, ..Control::default() })
}

/// How often the work under a [`Control::cancelled_once_started`] control looked at its flag:
/// the tool's checks, and the data writers' (not those of whoever runs it, before and after).
/// Always 0 for other controls.
pub fn looked(&self) -> u64 {
self.looked.load(Ordering::Relaxed)
}

/// Cancels a [`Control::cancelled_once_started`] control: called by [`crate::run_under`] once
/// the work is about to start.
pub(crate) fn cancel_if_started(&self) {
if self.cancel_at_start {
self.cancel();
}
}

/// The flag, looked at by the work: counted for a test's control only, so an ordinary run's
/// threads don't share a counter.
fn looked_at(&self) -> bool {
if self.cancel_at_start {
self.looked.fetch_add(1, Ordering::Relaxed);
}
self.is_cancelled()
}

/// Asks the tool to stop at its next check.
pub fn cancel(&self) {
self.cancel.store(true, Ordering::Relaxed);
Expand All @@ -55,7 +90,7 @@ impl Control {
/// their steps count here, and they stop when the run is cancelled.
impl rustgis_data::progress::Probe for Control {
fn cancelled(&self) -> bool {
self.is_cancelled()
self.looked_at()
}
fn begin(&self, total: u64) {
self.done.store(0, Ordering::Relaxed);
Expand All @@ -75,7 +110,7 @@ struct Progress {

impl rustgis_data::progress::Probe for Progress {
fn cancelled(&self) -> bool {
self.control.as_ref().is_some_and(|c| c.is_cancelled()) || rustgis_data::source::remote::check_still(&self.still).is_err()
self.control.as_ref().is_some_and(|c| c.looked_at()) || rustgis_data::source::remote::check_still(&self.still).is_err()
}
fn begin(&self, total: u64) {
if let Some(control) = &self.control {
Expand Down Expand Up @@ -145,7 +180,7 @@ impl Drop for Installed {

/// Whether the run on this thread was asked to stop.
pub(crate) fn cancelled() -> bool {
CURRENT.with(|c| c.borrow().as_ref().is_some_and(|c| c.is_cancelled()))
CURRENT.with(|c| c.borrow().as_ref().is_some_and(|c| c.looked_at()))
|| rustgis_data::source::remote::check_still(&rustgis_data::source::remote::current_still()).is_err()
}

Expand Down Expand Up @@ -224,6 +259,26 @@ pub(crate) fn advance(n: usize) {
mod tests {
use super::*;

/// A control cancelled once its run starts passes the check before the work, is cancelled for
/// the work's own looks, and counts them: work that looks stops, work that doesn't is seen.
#[test]
fn a_control_cancelled_once_started_counts_the_works_looks() {
let control = Control::cancelled_once_started();
let looks = crate::run_under("Looks", &control, || {
check()?;
Ok::<_, String>(())
});
assert!(looks.unwrap_err().contains("Looks was cancelled"));
assert_eq!(control.looked(), 1);
let blind = Control::cancelled_once_started();
let res = crate::run_under("Blind", &blind, || Ok::<_, String>(()));
assert!(res.is_err(), "cancelled all the same, by whoever ran it");
assert_eq!(blind.looked(), 0, "but the work never looked");
// An ordinary control isn't cancelled by starting.
let plain = Control::new();
assert!(crate::run_under("Plain", &plain, || Ok::<_, String>(())).is_ok());
}

/// Revocation during real file construction must reach the data writer's final rename,
/// even though the person did not press Cancel. Its prior file and the next run survive.
#[cfg(not(target_arch = "wasm32"))]
Expand Down
1 change: 1 addition & 0 deletions crates/analysis/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -696,6 +696,7 @@ pub fn run_under<T>(name: &str, control: &std::sync::Arc<control::Control>, work
let _data = control::install_progress(Some(control.clone()));
// No count until the work starts one: a control used before doesn't show that run's.
control::begin(0);
control.cancel_if_started();
let res = work();
rustgis_data::source::remote::check_still(&still)?;
if !control.is_cancelled() {
Expand Down
64 changes: 46 additions & 18 deletions crates/analysis/src/tools.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1576,25 +1576,31 @@ fn part(g: &Geometry, mask: &MultiPolygon<f64>, inside: bool) -> Option<Geometry

/// Clip each input feature to the union of only the selected clip polygons whose projected
/// extents meet it. Union candidates together so overlapping masks never duplicate output.
/// Cancel is looked at, and progress counted, mask by mask and feature by feature; one mask's
/// union, and one feature's overlay, can't be stopped once started (#1562).
fn clip(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
let ds = input(inp, p, "input")?;
let clip_ds = input(inp, p, "clip")?;
if clip_ds.geometry_type != GeometryType::Polygon {
return Err(format!("'{}' is not a polygon layer", clip_ds.name));
}
let masks: Vec<_> = rows(inp, p, "clip", clip_ds)
.into_iter()
.filter_map(|r| clip_ds.features.get(r)?.geometry.as_ref().and_then(to_multipolygon))
.map(|m| {
let ext = merc_extent(&m);
// Union even one polygon: collapsed rings have no filled area. Keep the original
// bounds too: rounding an edge must not hide a line the boolean overlay clips to it.
let mask = union_all(m.0.iter());
let ext = ext.union(&merc_extent(&mask));
(mask, ext)
})
.filter(|(m, _)| !m.0.is_empty())
.collect();
let mut masks = Vec::new();
let mask_rows = rows(inp, p, "clip", clip_ds);
crate::control::begin(mask_rows.len());
for (done, r) in mask_rows.into_iter().enumerate() {
// Each mask is a union: Cancel is looked at, and progress counted, before each.
crate::control::check()?;
crate::control::reached(done);
let Some(m) = clip_ds.features.get(r).and_then(|f| f.geometry.as_ref()).and_then(to_multipolygon) else { continue };
let ext = merc_extent(&m);
// Union even one polygon: collapsed rings have no filled area. Keep the original
// bounds too: rounding an edge must not hide a line the boolean overlay clips to it.
let mask = union_all(m.0.iter());
let ext = ext.union(&merc_extent(&mask));
if !mask.0.is_empty() {
masks.push((mask, ext));
}
}
let index = ExtentIndex::new(masks.iter().map(|(_, ext)| *ext));
// The masks reaching each pole: a point at one is in them at any longitude (#1033).
let at_poles = AtPoles::new(masks.iter().map(|(m, _)| plane_poles(m)));
Expand All @@ -1603,7 +1609,11 @@ fn clip(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
const MAX_CACHED_MASKS: usize = 8;
let mut cached_masks: std::collections::VecDeque<(Vec<usize>, MultiPolygon<f64>)> = std::collections::VecDeque::new();
let mut out = copy_schema(ds, out_name(p, format!("{}_Clip", ds.name)), ds.geometry_type);
for r in rows(inp, p, "input", ds) {
let input_rows = rows(inp, p, "input", ds);
crate::control::begin(input_rows.len());
Comment thread
pick2510 marked this conversation as resolved.
Resolved
for (done, r) in input_rows.into_iter().enumerate() {
crate::control::check()?;
crate::control::reached(done);
let Some(f) = ds.features.get(r) else { continue };
let Some(g) = f.geometry.as_ref() else { continue };
let ext = projected_extent(g);
Expand Down Expand Up @@ -2145,6 +2155,9 @@ fn split(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
/// of the (selected) polygons holding it, with the value, `count` (the polygons merged) and the
/// statistics asked for, as `<STAT>_<field>`, over exactly the records merged: a record without a
/// polygon is left out of both.
/// Dissolve: Cancel is looked at feature by feature while the groups are made, and group by group
/// while each is merged. A group's merge is one union of all its polygons, which can't be stopped
/// once it has started: with no field, the whole layer is one group (#1562).
fn dissolve(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
let ds = input(inp, p, "input")?;
if ds.geometry_type != GeometryType::Polygon {
Expand All @@ -2156,7 +2169,11 @@ fn dissolve(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
// Each group's key, polygons and records, in the order first met; found by its key's hash.
let mut groups: Vec<(Value, Vec<MultiPolygon<f64>>, Vec<usize>)> = Vec::new();
let mut index: HashMap<crate::stats::KeyPart<'_>, usize> = HashMap::new();
for r in each_once(rows(inp, p, "input", ds)) {
let input_rows = each_once(rows(inp, p, "input", ds));
crate::control::begin(input_rows.len());
for (done, r) in input_rows.into_iter().enumerate() {
crate::control::check()?;
crate::control::reached(done);
Comment thread
pick2510 marked this conversation as resolved.
Resolved
let Some(mp) = ds.features.get(r).and_then(|f| f.geometry.as_ref()).and_then(to_multipolygon) else { continue };
let value = fi.map_or(&Value::Null, |fi| ds.value(r, fi));
// A NaN equals no value, so each one stays a group of its own; −0 is 0, as it compares.
Expand All @@ -2183,7 +2200,11 @@ fn dissolve(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
fields.extend(added);
let mut out = Dataset::new(out_name(p, format!("{}_Dissolve", ds.name)), GeometryType::Polygon, fields);
let mut rounded = Rounded::new(requests.len());
for (key, parts, mut records) in groups {
// The unions are the work: counted, and Cancel looked at, group by group.
crate::control::begin(groups.len());
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
for (done, (key, parts, mut records)) in groups.into_iter().enumerate() {
crate::control::check()?;
crate::control::reached(done);
let n = parts.len() as i64;
let u = union_all(parts.iter().flat_map(|m| m.0.iter()));
let mut attrs = if fi.is_some() { vec![key, Value::Int(n)] } else { vec![Value::Int(n)] };
Expand Down Expand Up @@ -5665,7 +5686,11 @@ fn spatial_join(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
let join_rows = rows(inp, p, "join", join);

let mut join_geoms: Vec<Located<'_>> = Vec::with_capacity(join_rows.len());
for &jr in &join_rows {
// The join features converted: Cancel looked at, and progress counted, feature by feature.
crate::control::begin(join_rows.len());
for (done, &jr) in join_rows.iter().enumerate() {
crate::control::check()?;
crate::control::reached(done);
Comment thread
pick2510 marked this conversation as resolved.
Resolved
if let Some(f) = join.features.get(jr)
&& let Some(g) = f.geometry.as_ref()
{
Expand Down Expand Up @@ -5710,7 +5735,10 @@ fn spatial_join(inp: &dyn Inputs, p: &Json) -> Result<ToolResult, String> {
let mut written_features = 0usize;
let mut cand_buf = Vec::new();

for tr in target_rows {
crate::control::begin(target_rows.len());
for (done, tr) in target_rows.into_iter().enumerate() {
crate::control::check()?;
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
Comment thread
pick2510 marked this conversation as resolved.
Resolved
crate::control::reached(done);
let Some(tf) = target.features.get(tr) else { continue };
// The matches, in the order they're joined, with their distances when measured.
let mut matches: Vec<(usize, Option<f64>)> = Vec::new();
Expand Down
2 changes: 2 additions & 0 deletions crates/engine/src/cmd/cube.rs
Original file line number Diff line number Diff line change
Expand Up @@ -766,6 +766,8 @@ impl TrendRun {
let slopes = n.saturating_mul(n).saturating_mul(4).max(1);
let threads = rustgis_data::threads::max().min(1 + rustgis_analysis::scratch_budget() / slopes);
let part = cells.div_ceil(threads).max(1);
// No more threads than there are parts (10 cells on 8 threads make 5).
let threads = cells.div_ceil(part);
// The parts are taken from a list, by the threads started and by this one, so a
// thread the OS refuses leaves no part undone (and costs speed, not the run).
let parts = std::sync::Mutex::new(block.chunks(part * n).zip(out.chunks_mut(part * OUTPUTS)).enumerate().collect::<Vec<_>>());
Expand Down
2 changes: 1 addition & 1 deletion crates/engine/src/cmd/exports.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ type Start = fn(&Session, &Value) -> Result<Work, String>;

/// The exports the app writes on another thread ([`export_start`]), each with what it is called
/// (in its progress, and a cancelled run's error) and what it reads of the session.
const IN_BACKGROUND: &[(&str, &str, Start)] = &[
pub(crate) const IN_BACKGROUND: &[(&str, &str, Start)] = &[
("share.exportLayer", "Export Features", super::share::layer_work),
("share.exportRaster", "Export Raster", super::raster::export_work),
("share.exportMap", "Export Map", super::share::map_work),
Expand Down
2 changes: 1 addition & 1 deletion crates/engine/src/cmd/workflows.rs
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ type Start = fn(&Session, &Value) -> Result<Work, String>;
/// what it is called (in its progress, and a cancelled run's error) and what it reads of the
/// session: their work grows with the raster, and most of their passes spread it over the cores
/// from that thread. Run again with `{"ran": id}`, each adds the raster made (#1291 step 3).
const IN_BACKGROUND: &[(&str, &str, Start)] = &[
pub(crate) const IN_BACKGROUND: &[(&str, &str, Start)] = &[
("analysis.viewshed", "Viewshed", viewshed_work),
("analysis.interpolate", "Interpolate (IDW)", interpolate_work),
("analysis.suitability", "Suitability Modeler", suitability_work),
Expand Down
Loading
Loading