Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 31 additions & 8 deletions crates/engine/src/cmd/exports.rs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ pub(crate) const IN_BACKGROUND: &[(&str, &str, Start)] = &[
("share.exportRaster", "Export Raster", super::raster::export_work),
("share.exportMap", "Export Map", super::share::map_work),
("layout.export", "Export Layout", super::layout::export_work),
("report.export", "Export Report", super::report::export_work),
];

/// `command` and what it is called, when the app writes its file on another thread (see
Expand Down Expand Up @@ -125,14 +126,7 @@ pub(crate) fn drawn(
let written_path = written.is_ok().then(|| path.clone());
let answer = written.as_ref().map_or(Value::Null, Value::clone);
let then: Then = Box::new(move |s: &mut Session, _: &mut Value| {
use rustgis_data::diagnostics::Severity;
for (severity, text) in notices {
match severity {
Severity::Error => s.notify(text, true),
Severity::Warning => s.warn(text),
Severity::Info => s.notify(text, false),
}
}
say(s, notices);
let mut missing = Vec::new();
for (instance, name, error, copied) in failed {
let live = s.project.maps.iter_mut().flat_map(|m| m.layers.iter_mut()).find(|l| l.instance_id() == instance);
Expand Down Expand Up @@ -167,6 +161,35 @@ pub(crate) fn drawn(
})
}

/// What drawing an export on `copy` (a copy of the session) said, each notice with its severity,
/// to say again where the export lands; past the notices a session keeps, how many earlier ones
/// there were. `path` names the file in that count.
pub(crate) fn copy_notices(copy: &mut Session, path: &str) -> Vec<(rustgis_data::diagnostics::Severity, String)> {
let kept = std::mem::take(&mut copy.notices);
let earlier = copy.notices_given().saturating_sub(kept.len() as u64);
let mut said = Vec::new();
if earlier > 0 {
said.push((
rustgis_data::diagnostics::Severity::Warning,
format!("{path}: {earlier} earlier notices from drawing it aren't shown"),
));
}
said.extend(kept.into_iter().map(|n| (n.severity, n.text)));
said
}

/// Says `notices` (what an export said on a copy of the session) in `s`, each as it was said.
pub(crate) fn say(s: &mut Session, notices: Vec<(rustgis_data::diagnostics::Severity, String)>) {
use rustgis_data::diagnostics::Severity;
for (severity, text) in notices {
match severity {
Severity::Error => s.notify(text, true),
Severity::Warning => s.warn(text),
Severity::Info => s.notify(text, false),
}
}
}

#[cfg(not(target_arch = "wasm32"))]
/// An export ([`IN_BACKGROUND`]) started to write its file on another thread ([`ExportRun::run`])
/// while the session goes on: its inputs are taken now (cheaply: the features and pixels are
Expand Down
38 changes: 36 additions & 2 deletions crates/engine/src/cmd/labels.rs
Original file line number Diff line number Diff line change
Expand Up @@ -642,15 +642,46 @@ fn distance_range(s: &mut Session, p: &Value) -> CmdResult {
})
}

/// Whether layer `l` has text to convert into annotations at `scale`: a class's label field, or a
/// text expression. Decided on its fields, so before a layer read in place is read whole.
pub(crate) fn has_label_text(l: &rustgis_map::Layer, scale: f64) -> bool {
l.labels.in_order().any(|c| {
l.dataset.field_index(&c.field).is_some()
|| rustgis_map::label_by::LabelExprs::new(&c.by, &l.dataset, Some(scale)).is_some_and(|e| e.has_text())
})
}

/// Converts a layer's labels into an annotation layer: one point per label, at the label point,
/// carrying the label text and drawn with the same text symbol. Each class labelling a feature
/// gives it a point (a hidden class none); with classes besides Default, a CLASS field keeps
/// which, and the annotation has the same classes, each picking its points by it. The layer's
/// own labels turn off.
/// own labels turn off. A layer read in place converts all of its source's features, not just
/// those of the view, and says the rows it couldn't read; it stays read in place. The app reads
/// them ahead on a thread of its own, as for Load All Features, whatever asks for the conversion
/// there; a script or the command line reads them where the command runs. The conversion itself
/// runs on the app's thread: it adds the annotation layer and turns the layer's labels off in one
/// undoable step, a pass over features already in memory.
fn convert(s: &mut Session, p: &Value) -> CmdResult {
let id = s.layer_id(p)?;
let scale = s.map().scale();
let l = s.map().layer(id).ok_or("no such layer")?;
// Refused before a layer read in place is read whole for nothing.
if !has_label_text(s.map().layer(id).ok_or("no such layer")?, scale) {
return Err("the layer has no label field".into());
}
// Its features, all of them: a copy of a layer read in place holds those of its whole source.
let whole = if s.map().layer(id).ok_or("no such layer")?.is_streamed() {
let (ds, unread) = super::whole::take_or_read(s, id)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (performance): a streamed layer's whole source is read before the command checks it has any labels to convert.

The whole read runs here (and, in the app, on the read-ahead thread from reads.rs:102). The check at line 681 (the layer has no label field) comes after it. A layer with no label settings therefore reads its entire source, possibly a large PostGIS table, and then fails with that error. The read result is discarded.

Failure scenario: a PostGIS layer read in place with no label field set. Clicking Convert starts a full background read, then reports "the layer has no label field".

Suggested fix: move the label-field check onto the live layer before take_or_read, using the same fields/by_text logic. Factor it into a helper, such as has_label_field(l: &Layer), and call it in reads::prepare for labels.convert before whole::ahead, so no read is queued either. Add a test that a streamed layer with no labels fails without reading its source.

Confidence: high (about 85%).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6534525: convert checks first that the layer has label text (a class's label field or a text expression, decided on its fields by has_label_text) and refuses before any whole read. The app's read-ahead (reads::prepare) checks the same and makes no read when there's nothing to convert.

let mut copy = s.map().layer(id).ok_or("no such layer")?.clone();
copy.dataset = ds;
copy.source = None;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AGENTS.md, Never hold the window (about 55% confidence). This read is new in this PR, and the conversion loop that follows (line 724, for (row, ft) in l.dataset.features.iter()) now runs over every feature of a layer read in place, not just the view. In the app it runs in land on the UI thread, after the whole read has finished on its own thread. For a large PostGIS or web source that is one long pass with no Cancel check and no progress count, and the window is held for its whole length.

The doc comment's reason ("a pass over features already in memory") explains why the step must run on the app's thread, but not how long it can take. Options: build ds and moved from the copied layer on the read thread, so the UI thread only adds the layer and turns the labels off. Or, if the pass stays here, check Cancel and report progress every N features and say so in the doc comment.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The read, the part that grows with the source and touches files or the network, runs on a thread of its own in the app. The loop that follows is the same pass labels.convert already makes on the app's thread for an in-memory layer of any size, and the doc comment says why it stays there: one undoable step that adds the layer and turns its labels off. Moving the conversion itself off the frame would change it for in-memory layers too, which is outside this fix.

Some((copy, unread))
} else {
None
};
let l = match &whole {
Some((copy, _)) => copy,
None => s.map().layer(id).ok_or("no such layer")?,
};
let fields: Vec<Option<usize>> = l.labels.in_order().map(|c| l.dataset.field_index(&c.field)).collect();
// Its classes' expressions, as the map draws them now (at its scale).
let exprs: Vec<Option<rustgis_map::label_by::LabelExprs>> =
Expand Down Expand Up @@ -793,5 +824,8 @@ fn convert(s: &mut Session, p: &Value) -> CmdResult {
if let Some(note) = cut_note {
s.notify(note, false);
}
if let Some((_, unread)) = &whole {
unread.say(s);
}
Ok(json!({"layer": new_id, "annotations": n}))
}
34 changes: 4 additions & 30 deletions crates/engine/src/cmd/layer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2985,23 +2985,12 @@ fn define_projection(s: &mut Session, p: &Value) -> CmdResult {
/// Loads a streamed layer's features into memory, so it can be edited like any other layer.
fn load_all(s: &mut Session, p: &Value) -> CmdResult {
let id = s.layer_id(p)?;
let key = load_all_key(s.map().layer(id).ok_or("no such layer")?);
let ahead = super::reads::take::<LoadedAll>(s, &key);
let (mut ds, unread) = super::whole::take_or_read(s, id)?;
let l = s.map().layer(id).ok_or("no such layer")?;
let (mut ds, unread) = match ahead {
// Read (or failed) from the source the layer still reads, listing the fields it lists
// now: one opened afresh since, or with other fields, is read again.
Some((copied, fields, read)) if l.source.as_ref().is_some_and(|src| src.same_opening(&copied)) && fields == l.dataset.fields => {
read?
}
_ => l.load_all_counted()?,
};
// Named as the layer is now: it may have been renamed while the read ran.
ds.name = l.dataset.name.clone();
let (n, name) = (ds.features.len(), l.name.clone());
if unread > 0 {
s.warn(super::whole::unread_notice(&name, unread));
}
let n = ds.features.len();
unread.say(s);
let l = s.map_mut().layer_mut(id).ok_or("no such layer")?;
l.dataset = ds;
l.dataset.windowed = false;
Expand All @@ -3011,25 +3000,10 @@ fn load_all(s: &mut Session, p: &Value) -> CmdResult {
Ok(json!({"layer": id, "features": n}))
}

/// Load All's read made before it runs: the source it read, the fields the layer listed then, and
/// what it read with the number of rows that couldn't be read.
type LoadedAll = (rustgis_map::source::SourceLayer, Vec<rustgis_data::Field>, Result<(rustgis_data::Dataset, usize), String>);

/// The key Load All's read of `l` is taken by (see [`super::reads`]).
fn load_all_key(l: &rustgis_map::Layer) -> String {
format!("loadAll:{}", l.instance_id())
}

/// Load All's read of the streamed layer `p` names, made before it runs ([`super::reads`]), on a
/// copy of the layer.
pub(crate) fn load_all_ahead(s: &Session, p: &Value) -> Option<super::reads::ReadWork> {
let l = s.map().layer(s.layer_id(p).ok()?)?;
let mut copy = l.clone();
let source = copy.source.as_mut()?;
// Its own read: the live layer keeps the one under way.
source.detach_read();
let (copied, fields) = (source.clone(), l.dataset.fields.clone());
Some(super::reads::ahead("Load All Features", load_all_key(l), move || (copied, fields, copy.load_all_counted())))
super::whole::ahead(s, p, "Load All Features")
}

/// Adds a layer file: a layer with its data, symbology and labels.
Expand Down
13 changes: 2 additions & 11 deletions crates/engine/src/cmd/layout.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3769,17 +3769,8 @@ pub(crate) fn export_work(s: &Session, p: &Value) -> Result<super::exports::Work
Ok(json!({"path": path, "bytes": bytes.len()}))
})();
// What drawing it said (frames drawn as pictures, fonts it lacks), as the command says it
// here; past the notices a session keeps, how many earlier ones there were.
let kept = std::mem::take(&mut copy.notices);
let earlier = copy.notices_given().saturating_sub(kept.len() as u64);
let mut said: Vec<_> = Vec::new();
if earlier > 0 {
said.push((
rustgis_data::diagnostics::Severity::Warning,
format!("{path}: {earlier} earlier notices from drawing it aren't shown"),
));
}
said.extend(kept.into_iter().map(|n| (n.severity, n.text)));
// here.
let said = super::exports::copy_notices(&mut copy, &path);
super::exports::drawn(&copy.project.maps, said, written, path)
}))
}
Expand Down
15 changes: 12 additions & 3 deletions crates/engine/src/cmd/reads.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
//! Reads the desktop app makes on a thread of its own before a command adds what they read
//! (#1291 step 3): Add Data's whole-file reads (a raster, a raster GDAL converts, a GeoJSON,
//! delimited text or spreadsheet read whole, a CityJSON or CityGML model's layers), Reload Data, Load All Features, native models or scripts and
//! the Error Inspector's topology check (#1291). [`read_ahead`]
//! delimited text or spreadsheet read whole, a CityJSON or CityGML model's layers), Reload Data, Load All
//! Features, Convert To Annotation's read of a layer read in place, native models or scripts and the
//! Error Inspector's topology check (#1291). [`read_ahead`]
//! takes what the read needs of the session, [`ReadAhead::run`] reads on the thread, and [`read_land`]
//! runs the admitted command under its initiating owner: the command takes
//! it instead of reading again, so what it adds, says, logs and puts in Undo is the same as when
Expand Down Expand Up @@ -98,6 +99,14 @@ pub(crate) fn prepare(s: &Session, op: Authorized<CommandOp>, owner: Continuatio
let layer = s.map().layer(s.layer_id(params)?).ok_or("no such layer")?;
("Load All Features", super::layer::load_all_ahead(s, params), layer.source.clone())
}
"labels.convert" => {
let layer = s.map().layer(s.layer_id(params)?).ok_or("no such layer")?;
// Nothing to convert: the command says so at once, with no read.
let work = super::labels::has_label_text(layer, s.map().scale())
.then(|| super::whole::ahead(s, params, "Convert To Annotation"))
.flatten();
("Convert To Annotation", work, layer.source.clone())
}
"layer.reloadData" => {
#[cfg(not(target_arch = "wasm32"))]
let work = super::repair::reload_ahead(s, params);
Expand Down Expand Up @@ -130,7 +139,7 @@ pub(crate) fn prepare(s: &Session, op: Authorized<CommandOp>, owner: Continuatio
(name, work, None)
}
"topology.validate" => ("Error Inspector", super::design::topology_ahead(s, params), None),
_ => return Err("only Add Data, Reload Data, Load All, tile archives, models, scripts and topology prepare background work".into()),
_ => return Err("only Add Data, Load All, Reload Data, Convert To Annotation, models, scripts, tile archives and topology prepare background work".into()),
};
let name = work.as_ref().map_or(name, |work| work.name);
Ok(ReadAhead { name, op, owner, source, work })
Expand Down
Loading
Loading