Repository navigation
fix(data): convert whole streamed layers off the UI thread #1889
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -1596,43 +1596,130 @@ fn similar(s: &mut Session, p: &Value) -> CmdResult { | |||||||||||||||||
| Ok(json!({"similar": rows})) | ||||||||||||||||||
| } | ||||||||||||||||||
|
|
||||||||||||||||||
| /// Converts a layer to GeoJSON, CSV, KML, GPX or WKT text, written to `path` or returned; says when | ||||||||||||||||||
| /// the layer's Z or M values were dropped on import, its positions are approximate or aren't | ||||||||||||||||||
| /// longitude/latitude, or CSV, KML or GPX couldn't hold some of it. | ||||||||||||||||||
| /// Converts text on the caller's thread for scripts and browser downloads. A streamed layer | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Performance (low-medium, ~50%): the text path still runs on the UI thread in the desktop app.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. #1857 explicitly permits the no-path text result to stay synchronous. Preserving the immediate |
||||||||||||||||||
| /// needs a file path so its full source can be read off the desktop frame, rather than returning | ||||||||||||||||||
| /// only its loaded view. File conversions use the shared export lifecycle. | ||||||||||||||||||
| fn convert(s: &mut Session, p: &Value) -> CmdResult { | ||||||||||||||||||
| if ps(p, "path").is_some() { | ||||||||||||||||||
| return super::exports::write_here(s, p, "data.convert"); | ||||||||||||||||||
| } | ||||||||||||||||||
| let id = s.layer_id(p)?; | ||||||||||||||||||
| if s.map().layer(id).is_some_and(|l| l.is_streamed()) { | ||||||||||||||||||
| return Err("a streamed layer needs a path to convert its whole source; text conversion would contain only the loaded view".into()); | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bug (medium, ~65%): the browser can no longer convert a streamed layer. The browser has one thread and is out of scope for the background rule, and streamed sources exist on wasm (
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This follows #1857’s Expected section: text-only conversion may refuse a streamed layer instead of returning a partial view. Browser downloads use that text path, so they now report the refusal rather than silently omit features. The Data Interop table in docs/user-guide/data-formats.md already states that streamed text requires a path. I have also made the browser limitation explicit in the PR description; fully loaded browser layers still convert. Whole-source browser downloads would extend the requested scope. |
||||||||||||||||||
| } | ||||||||||||||||||
| let ds = feature_ds(s, p)?; | ||||||||||||||||||
| let (text, notes) = converted(ds, p)?; | ||||||||||||||||||
| let null_numbers = rustgis_data::export::numbers_written_as_null(&ds.features, &ds.fields, conversion_format(p)?); | ||||||||||||||||||
| let target = ds.name.clone(); | ||||||||||||||||||
| for note in notes { | ||||||||||||||||||
| s.notify(note, false); | ||||||||||||||||||
| } | ||||||||||||||||||
| super::share::notify_null_numbers(s, &target, null_numbers); | ||||||||||||||||||
| Ok(json!({"text": text})) | ||||||||||||||||||
| } | ||||||||||||||||||
|
|
||||||||||||||||||
| /// Captures a conversion's layer and parameters without reading its source on the UI thread. | ||||||||||||||||||
| /// GeoJSON and CSV use the chunk writer; KML, GPX and WKT read whole within the existing memory | ||||||||||||||||||
| /// limits, then check Cancel between features (one geometry's serialization is indivisible). | ||||||||||||||||||
|
Comment on lines
+1621
to
+1623
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. AGENTS.md (confidence ~70%): the whole-source read for KML, GPX and WKT is an undocumented long step without a Cancel check. For a streamed layer, the Suggested wording (replaces these three doc lines):
Suggested change
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed that Cancel waits for the existing whole-source read. The current doc comment already identifies that boundary: KML, GPX and WKT ‘read whole within the existing memory limits, then check Cancel between features’. It also identifies one geometry’s serialization as indivisible. That states the long read before the per-feature checks; this is not an undocumented step or a change in cancellation behavior. I’m leaving the proposed more explicit wording for the release quality pass. |
||||||||||||||||||
| pub(crate) fn convert_work(s: &Session, p: &Value) -> Result<super::exports::Work, String> { | ||||||||||||||||||
| feature_ds(s, p)?; | ||||||||||||||||||
| let id = s.layer_id(p)?; | ||||||||||||||||||
| let layer = s.map().layer(id).ok_or("no such layer")?.clone(); | ||||||||||||||||||
| let path = ps(p, "path").ok_or("path is required")?.to_string(); | ||||||||||||||||||
| conversion_format(p)?; | ||||||||||||||||||
| let params = p.clone(); | ||||||||||||||||||
| Ok(Box::new(move || { | ||||||||||||||||||
| let format = conversion_format(¶ms)?; | ||||||||||||||||||
| let chunks = if layer.is_streamed() && matches!(format, "geojson" | "csv") { layer.chunk_export()? } else { None }; | ||||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Possible overwrite of the layer's own source (low confidence, ~25%).
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is an explicit whole-file replacement, rather than an in-place GeoPackage table update. |
||||||||||||||||||
| let (mut notes, null_numbers) = if let Some((source, ds)) = chunks { | ||||||||||||||||||
| let result = rustgis_data::export::write_source_as_in( | ||||||||||||||||||
| source.as_ref(), | ||||||||||||||||||
| &ds, | ||||||||||||||||||
| std::path::Path::new(&path), | ||||||||||||||||||
| rustgis_data::raster::Crs::LonLat, | ||||||||||||||||||
| format, | ||||||||||||||||||
| ) | ||||||||||||||||||
| .map_err(|e| e.to_string())?; | ||||||||||||||||||
| let mut notes = rustgis_data::export_notes([(&ds, layer.extent())], &path, rustgis_data::raster::Crs::LonLat); | ||||||||||||||||||
| if format == "csv" { | ||||||||||||||||||
| notes.extend(result.csv.notes(&path)); | ||||||||||||||||||
| } | ||||||||||||||||||
| if result.skipped > 0 { | ||||||||||||||||||
| notes.push(format!("{path}: {} rows couldn't be read and were left out", result.skipped)); | ||||||||||||||||||
| } | ||||||||||||||||||
| notes.extend( | ||||||||||||||||||
| rustgis_data::source::unread_dates_note(&ds.fields, &result.unread_dates) | ||||||||||||||||||
| .map(|n| format!("{path}: {}: {n}, so they're empty in the export", ds.name)), | ||||||||||||||||||
| ); | ||||||||||||||||||
| (notes, result.null_numbers) | ||||||||||||||||||
| } else { | ||||||||||||||||||
| let (whole, unreadable) = if layer.is_streamed() { | ||||||||||||||||||
| layer.load_all_counted().map(|(ds, n)| (Some(ds), n))? | ||||||||||||||||||
| } else { | ||||||||||||||||||
| (None, rustgis_map::Unreadable::default()) | ||||||||||||||||||
| }; | ||||||||||||||||||
| let ds = whole.as_ref().unwrap_or(&layer.dataset); | ||||||||||||||||||
| rustgis_data::progress::begin(ds.features.len()); | ||||||||||||||||||
| let (text, mut notes) = converted(ds, ¶ms)?; | ||||||||||||||||||
| rustgis_data::io::write_atomic(std::path::Path::new(&path), text.as_bytes()).map_err(|e| format!("{path}: {e}"))?; | ||||||||||||||||||
| if unreadable.rows > 0 { | ||||||||||||||||||
| notes.push(format!("{path}: {} rows couldn't be read and were left out", unreadable.rows)); | ||||||||||||||||||
| } | ||||||||||||||||||
| notes.extend(unreadable.dates.iter().map(|n| format!("{path}: {}: {n}, so they're empty in the export", ds.name))); | ||||||||||||||||||
| (notes, rustgis_data::export::numbers_written_as_null(&ds.features, &ds.fields, format)) | ||||||||||||||||||
| }; | ||||||||||||||||||
| let bytes = std::fs::metadata(&path).map_err(|e| format!("{path}: {e}"))?.len(); | ||||||||||||||||||
| notes.push(format!("Exported {path}")); | ||||||||||||||||||
| let target = path.clone(); | ||||||||||||||||||
| let then = (null_numbers > 0).then(|| { | ||||||||||||||||||
| Box::new(move |s: &mut Session, _: &mut Value| { | ||||||||||||||||||
| super::share::notify_null_numbers(s, &target, null_numbers); | ||||||||||||||||||
| Ok(()) | ||||||||||||||||||
| }) as super::exports::Then | ||||||||||||||||||
| }); | ||||||||||||||||||
| Ok(super::exports::Wrote { | ||||||||||||||||||
| answer: json!({"path": path, "bytes": bytes}), | ||||||||||||||||||
| notes, | ||||||||||||||||||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||||||||||||||||||
| written_path: Some(path), | ||||||||||||||||||
| then, | ||||||||||||||||||
| #[cfg(not(target_arch = "wasm32"))] | ||||||||||||||||||
| owner: None, | ||||||||||||||||||
| }) | ||||||||||||||||||
| })) | ||||||||||||||||||
| } | ||||||||||||||||||
|
|
||||||||||||||||||
| /// Validates the existing case-sensitive format parameter before starting background work. | ||||||||||||||||||
| fn conversion_format(p: &Value) -> Result<&str, String> { | ||||||||||||||||||
| match ps(p, "format").ok_or("format is required")? { | ||||||||||||||||||
| f @ ("geojson" | "csv" | "kml" | "gpx" | "wkt") => Ok(f), | ||||||||||||||||||
| f => Err(format!("unknown format '{f}'")), | ||||||||||||||||||
| } | ||||||||||||||||||
| } | ||||||||||||||||||
|
|
||||||||||||||||||
| /// Serializes a complete dataset and collects its loss notices for delivery after success. | ||||||||||||||||||
| fn converted(ds: &Dataset, p: &Value) -> Result<(String, Vec<String>), String> { | ||||||||||||||||||
| let name_field = | ||||||||||||||||||
| ps(p, "nameField").and_then(|n| ds.field_index(n)).or_else(|| ds.fields.iter().position(|f| f.name.eq_ignore_ascii_case("name"))); | ||||||||||||||||||
| let text = match ps(p, "format").ok_or("format is required")? { | ||||||||||||||||||
| "geojson" => rustgis_data::io::to_geojson(ds).to_string(), | ||||||||||||||||||
| let format = conversion_format(p)?; | ||||||||||||||||||
| let text = match format { | ||||||||||||||||||
| "geojson" => rustgis_data::io::to_geojson_text(ds).map_err(|e| e.to_string())?, | ||||||||||||||||||
| "csv" => rustgis_data::io::to_csv(ds).map_err(|e| e.to_string())?, | ||||||||||||||||||
| "kml" => rustgis_data::interop::to_kml(ds, name_field).map_err(|e| e.to_string())?, | ||||||||||||||||||
| "gpx" => rustgis_data::interop::to_gpx(ds, name_field).map_err(|e| e.to_string())?, | ||||||||||||||||||
| "wkt" => rustgis_data::interop::to_wkt_csv(ds), | ||||||||||||||||||
| f => return Err(format!("unknown format '{f}'")), | ||||||||||||||||||
| _ => rustgis_data::interop::to_wkt_csv(ds).map_err(|e| e.to_string())?, | ||||||||||||||||||
| }; | ||||||||||||||||||
| // Every format here is longitude/latitude. | ||||||||||||||||||
| let target = ps(p, "path").unwrap_or(&ds.name); | ||||||||||||||||||
| let mut notes = rustgis_data::export_notes([(ds, ds.extent())], target, rustgis_data::raster::Crs::LonLat); | ||||||||||||||||||
| // CSV holds a point a row, GPX a name for each feature and no areas. | ||||||||||||||||||
| notes.extend(rustgis_data::io::export_losses_notes(ds, ps(p, "format").unwrap_or(""), name_field, target)); | ||||||||||||||||||
| notes.extend(rustgis_data::io::export_losses_notes(ds, format, name_field, target)); | ||||||||||||||||||
| // KML and GPX hold features with geometry only. | ||||||||||||||||||
| if let Some(ext @ ("kml" | "gpx")) = ps(p, "format") { | ||||||||||||||||||
| if let ext @ ("kml" | "gpx") = format { | ||||||||||||||||||
| let without = | ||||||||||||||||||
| if ext == "gpx" { rustgis_data::io::gpx_left_out(ds) } else { ds.features.iter().filter(|f| f.geometry.is_none()).count() }; | ||||||||||||||||||
| notes.extend(rustgis_data::io::left_out_note(target, ext, without)); | ||||||||||||||||||
| } | ||||||||||||||||||
| let out = match ps(p, "path") { | ||||||||||||||||||
| Some(path) => { | ||||||||||||||||||
| rustgis_data::io::write_atomic(std::path::Path::new(path), text.as_bytes()).map_err(|e| format!("{path}: {e}"))?; | ||||||||||||||||||
| json!({"path": path, "bytes": text.len()}) | ||||||||||||||||||
| } | ||||||||||||||||||
| None => json!({"text": text}), | ||||||||||||||||||
| }; | ||||||||||||||||||
| for note in notes { | ||||||||||||||||||
| s.notify(note, false); | ||||||||||||||||||
| } | ||||||||||||||||||
| Ok(out) | ||||||||||||||||||
| Ok((text, notes)) | ||||||||||||||||||
| } | ||||||||||||||||||
|
|
||||||||||||||||||
| /// Binds the extent layer, or all full-extent layers, without computing a view or loading pixels. | ||||||||||||||||||
|
|
||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Behaviour change for text callers (medium confidence, ~60%). The text returned by
data.convert(nopath) used to beto_geojson(ds).to_string(), with keys sorted (features,name,type). It is now this streaming writer, which emitstype,name,featuresin that order. The same features and values come out, but the bytes differ for scripts, the Pythontake_noticesconsumers, and the browser download. TheGeoJsonWriterdoc notes the order difference, but the PR description anddocs/user-guide/data-formats.mddon't mention it. Suggest a sentence in the docs, or a test that pins the output.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The GeoJSON object member order changes, while the feature order, values and parsed result remain the same. The command does not promise canonical JSON bytes. I’m leaving the documentation wording suggestion for the release quality pass; pinning incidental object order would constrain the shared cancellable writer without changing conversion semantics.