Repository navigation
feat(data): remember the file a layer was read from and say when they differ #1753
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
1cbc5dd
b2c4ab0
761b398
6aa5de8
0230eef
203ced5
c06cb93
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1401,8 +1401,11 @@ fn add_data(s: &mut Session, p: &Value) -> CmdResult { | |
| let src = rustgis_data::source::shp::ShpSource::open_path_in(Path::new(path), name, Some(crs)).map_err(|e| e.to_string())?; | ||
| return add_source_with(s, p, Origin::Path(path), Some(Box::new(src)), None); | ||
| } | ||
| let mut read = rustgis_data::io::read_shapefile_in(Path::new(path), name, Some(crs)).map_err(|e| e.to_string())?; | ||
| read.0.source = Some(path.to_string()); | ||
| // Read ahead in the desktop app ([`add_data_ahead`]). | ||
| let read = match super::reads::take(s, &whole_key(path)) { | ||
| Some(read) => read, | ||
| None => read_shapefile_with(path, name, crs), | ||
| }?; | ||
| return add_read(s, p, read, None); | ||
| } | ||
| // A large Shapefile is read in place, as a GeoPackage is; smaller ones are loaded whole. | ||
|
|
@@ -1580,19 +1583,72 @@ fn read_whole(p: &Value, path: &str) -> WholeRead { | |
| } | ||
| std::fs::read(path).map_err(|e| format!("{path}: {e}")) | ||
| })?; | ||
| match delimited { | ||
| Some(read) => Ok(read), | ||
| None => rustgis_data::io::read_path(Path::new(path)).map_err(|e| format!("{path}: {e}")), | ||
| if let Some(read) = delimited { | ||
| return Ok(read); | ||
| } | ||
| let read = || rustgis_data::io::read_path(Path::new(path)).map_err(|e| format!("{path}: {e}")); | ||
| // A file its edits can be written back to: the files as read, to compare with before then. | ||
| #[cfg(not(target_arch = "wasm32"))] | ||
| if rustgis_data::sync::is_syncable(path) { | ||
| return read_synced(path, read); | ||
| } | ||
| read() | ||
| } | ||
|
|
||
| /// A Shapefile read whole in coordinate system `crs`, which replaces its `.prj`, with the record | ||
| /// of what was read ([`read_synced`]). | ||
| #[cfg(not(target_arch = "wasm32"))] | ||
| fn read_shapefile_with(path: &str, name: &str, crs: rustgis_data::raster::Crs) -> WholeRead { | ||
| read_synced(path, || { | ||
| let mut read = rustgis_data::io::read_shapefile_in(Path::new(path), name, Some(crs)).map_err(|e| e.to_string())?; | ||
| read.0.source = Some(path.to_string()); | ||
| Ok(read) | ||
| }) | ||
| } | ||
|
|
||
| /// `read`, the read of the file at `path`, with the record of what it read: the files hashed | ||
| /// just before ([`rustgis_data::sync::FileSync::around`]) and the digest of the rows read. Its | ||
| /// work grows with the file, so the desktop app makes it off the UI thread ([`add_data_ahead`]). | ||
| /// When the files couldn't be hashed the layer has no record, and a note says it can't be | ||
| /// compared with its file later. | ||
| #[cfg(not(target_arch = "wasm32"))] | ||
| fn read_synced(path: &str, read: impl FnOnce() -> WholeRead) -> WholeRead { | ||
| // A file past what the readers take (a Shapefile's `.dbf` too) is refused by the read: not | ||
| // hashed first. | ||
| let over = |p: &Path| std::fs::metadata(p).is_ok_and(|m| m.len() > rustgis_data::io::MAX_BYTES); | ||
| let shp = path.to_ascii_lowercase().ends_with(".shp"); | ||
| if over(Path::new(path)) || (shp && over(&Path::new(path).with_extension("dbf"))) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Test coverage (AGENTS.md, Test both ways): the over-cap early return has no test. Two cases need a regression test:
A sparse file keeps this cheap:
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Leaving this without a test, as in the earlier thread on this branch. Exercising it needs a file over |
||
| return read(); | ||
| } | ||
| let (read, files) = rustgis_data::sync::FileSync::around(Path::new(path), read).map_err(|e| format!("{path}: {e}"))?; | ||
| let (mut ds, mut report) = read?; | ||
| // Kept whole, as the record checks it later: a relative path would be taken from wherever | ||
| // RustGIS runs then, not where the file was read. | ||
| if files.is_some() | ||
| && let Ok(full) = std::path::absolute(path) | ||
| { | ||
| ds.source = Some(full.display().to_string()); | ||
| } | ||
| report.synced = match files { | ||
| Some(files) => { | ||
| Some(rustgis_data::sync::Synced { files, rows: rustgis_data::sync::rows_digest(&ds).map_err(|e| format!("{path}: {e}"))? }) | ||
| } | ||
| None => { | ||
| report.messages.push(format!( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. AGENTS.md (Never silent, test both ways): the notice on a part that can't be hashed has no test. The "couldn't be read to record it" branch of
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added in 0230eef. The engine test adds a Shapefile with a folder where its |
||
| "{path}: a part of it couldn't be read to record it, so RustGIS can't tell later whether the file changed since; add it again to compare it" | ||
| )); | ||
| None | ||
| } | ||
| }; | ||
| Ok((ds, report)) | ||
| } | ||
|
|
||
| /// Add Data's read of the file `p` names, made before it runs ([`super::reads`]): a raster read | ||
| /// whole or converted by GDAL, a GeoJSON read whole, delimited text, a spreadsheet or a city model | ||
| /// (CityJSON or CityGML) read into its layers. `None` for | ||
| /// (CityJSON or CityGML) read into its layers, a Shapefile read whole. `None` for | ||
| /// any other file, which [`add_data`] reads as it adds it. Each is picked only where `add_data` | ||
| /// takes that read, so a read made is never wasted; a few it makes whole are left to it (a GeoJSON | ||
| /// with `"stream"` given, a `.json` with a coordinate system or layer given, a large GeoJSON that | ||
| /// isn't a FeatureCollection). | ||
| /// takes that read, so a read made is never wasted; a few it makes whole are left to it (a `.json` | ||
| /// with a coordinate system or layer given, a large GeoJSON that isn't a FeatureCollection). | ||
| #[cfg(not(target_arch = "wasm32"))] | ||
| pub(crate) fn add_data_ahead(p: &Value) -> Option<super::reads::ReadWork> { | ||
| use super::reads::ahead; | ||
|
|
@@ -1616,19 +1672,42 @@ pub(crate) fn add_data_ahead(p: &Value) -> Option<super::reads::ReadWork> { | |
| } | ||
| let given = |k: &str| ps(p, k).is_some_and(|v| !v.trim().is_empty()); | ||
| let lower = path.to_ascii_lowercase(); | ||
| // A GeoJSON read whole: small, with no "stream" given (which add_data decides by), and not a | ||
| // CityJSON model (named so, or a .json with a coordinate system or a layer given). | ||
| // A GeoJSON read whole: small with no "stream" given (which add_data decides by), or any size | ||
| // (within the reader's limit) with `"stream": false`; and not a CityJSON model (named so, or a | ||
| // .json with a coordinate system or a layer given). | ||
| let geojson = is_geojson(&path) | ||
| && !rustgis_data::cityjson::is_cityjson_name(&path) | ||
| && b(p, "stream").is_none() | ||
| && b(p, "stream") != Some(true) | ||
| && !(lower.ends_with(".json") && (given("crs") || given("layer"))) | ||
| && std::fs::metadata(&path).is_ok_and(|m| m.is_file() && m.len() <= rustgis_data::source::geojson::IN_PLACE_ABOVE); | ||
| && std::fs::metadata(&path) | ||
| .is_ok_and(|m| m.is_file() && (b(p, "stream") == Some(false) || m.len() <= rustgis_data::source::geojson::IN_PLACE_ABOVE)); | ||
| // A city model: its layers, read whole. A model named `.json` (not `.city.json`) is a file of | ||
| // several layers only with a coordinate system or a layer given, which Add Data reads as it runs. | ||
| if rustgis_data::io::is_multi_layer_name(&path) && is_city_model_path(&path) { | ||
| let (key, q) = (city_key(&path), p.clone()); | ||
| return Some(ahead(NAME, key, move || read_city_file(&q, &path))); | ||
| } | ||
| // A Shapefile read whole, in the coordinate system given if one is: as add_data reads it (one | ||
| // past the whole-file reader's cap, or with more records than it takes, is read in place). | ||
| if lower.ends_with(".shp") { | ||
| let q = past_cap_shapefile(&path, p).unwrap_or_else(|| p.clone()); | ||
| let stream = b(&q, "stream").unwrap_or_else(|| { | ||
| rustgis_data::source::shp::record_count(Path::new(&path)).is_some_and(|n| n > rustgis_map::source::STREAM_ABOVE) | ||
| }); | ||
| if stream { | ||
| return None; | ||
| } | ||
| let key = whole_key(&path); | ||
| return match ps(&q, "crs").map(str::trim).filter(|c| !c.is_empty()) { | ||
| Some(text) => { | ||
| // One add_data can't parse it refuses as it runs. | ||
| let crs = rustgis_data::raster::Crs::parse(text).ok()?; | ||
| let name = Path::new(&path).file_stem().and_then(|n| n.to_str()).unwrap_or("layer").to_string(); | ||
| Some(ahead(NAME, key, move || read_shapefile_with(&path, &name, crs))) | ||
| } | ||
| None => Some(ahead(NAME, key, move || read_whole(&q, &path))), | ||
| }; | ||
| } | ||
| let (key, q) = (whole_key(&path), p.clone()); | ||
| if rustgis_data::sheets::is_sheet_name(&path) { | ||
| return Some(ahead(NAME, key, move || read_sheet_file(&q, &path))); | ||
|
|
@@ -4731,7 +4810,11 @@ pub(crate) fn add_read( | |
| } | ||
| let n = ds.features.len(); | ||
| let empty_before = super::map_is_empty(s); | ||
| let synced = report.synced; | ||
| let id = add_dataset(s, ds); | ||
| if let Some(l) = s.map_mut().layer_mut(id) { | ||
| l.synced = synced; | ||
| } | ||
| if zoom.unwrap_or(empty_before) | ||
| && let Some(e) = s.map().layer(id).map(|l| l.extent()) | ||
| { | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,133 @@ | ||||||||||||||||
| //! A layer kept in the project and the file it was read from: whether its edits and the file are | ||||||||||||||||
| //! in step (#1385). | ||||||||||||||||
|
|
||||||||||||||||
| use crate::{CmdResult, CommandSpec as C, Session}; | ||||||||||||||||
| use serde_json::Value; | ||||||||||||||||
|
|
||||||||||||||||
| pub fn commands() -> Vec<C> { | ||||||||||||||||
| vec![ | ||||||||||||||||
| C::new("source.syncState", "Source File State", "Data › Data Source", sync_state).roles(&[]).effects(crate::policy::Effects::NONE.with(crate::policy::Effect::Files)).view().params( | ||||||||||||||||
| r#"{"layer"?: "name|id (default: the active layer): one read whole from a Shapefile or .geojson file"} — whether it has edits its file lacks ("edited"), the file changed since it was read ("sourceChanged"), both ("both") or neither ("inSync"), and which of the file's parts changed"#, | ||||||||||||||||
| ), | ||||||||||||||||
| ] | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| /// The key a check of layer `id`'s files made ahead is taken by (see [`super::reads`]). | ||||||||||||||||
| #[cfg(not(target_arch = "wasm32"))] | ||||||||||||||||
| fn ahead_key(id: u32) -> String { | ||||||||||||||||
| format!("syncState:{id}") | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| /// A check of a layer made ahead: the record, path and rows checked (the rows shared with the | ||||||||||||||||
| /// layer, not copied), the rows' digest and what changed in the files. | ||||||||||||||||
| #[cfg(not(target_arch = "wasm32"))] | ||||||||||||||||
| struct Checked { | ||||||||||||||||
| synced: rustgis_data::sync::Synced, | ||||||||||||||||
| path: String, | ||||||||||||||||
| dataset: rustgis_data::Dataset, | ||||||||||||||||
| rows: Result<u64, String>, | ||||||||||||||||
| changed: Result<Vec<String>, String>, | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| #[cfg(not(target_arch = "wasm32"))] | ||||||||||||||||
| impl Checked { | ||||||||||||||||
| /// Whether it checked layer `l` as it is: the same record and path, and the same rows (the | ||||||||||||||||
| /// same allocation: an edit copies them), fields and coordinate system. | ||||||||||||||||
| fn of(&self, l: &rustgis_map::Layer) -> bool { | ||||||||||||||||
| l.synced.as_ref() == Some(&self.synced) | ||||||||||||||||
| && l.dataset.source.as_deref() == Some(self.path.as_str()) | ||||||||||||||||
| && l.dataset.features.shares(&self.dataset.features) | ||||||||||||||||
| && l.dataset.fields == self.dataset.fields | ||||||||||||||||
| && l.dataset.crs == self.dataset.crs | ||||||||||||||||
| } | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| /// The check `source.syncState` makes before it runs in the desktop app: every part of the | ||||||||||||||||
| /// file read and hashed, and the layer's rows hashed, on a thread of its own with progress and | ||||||||||||||||
| /// Cancel. | ||||||||||||||||
| #[cfg(not(target_arch = "wasm32"))] | ||||||||||||||||
| pub(crate) fn sync_ahead(s: &Session, p: &Value) -> Option<super::reads::ReadWork> { | ||||||||||||||||
| let id = s.layer_id(p).ok()?; | ||||||||||||||||
| let l = s.map().layer(id)?; | ||||||||||||||||
| let (synced, path, dataset) = (l.synced.clone()?, l.dataset.source.clone().filter(|p| local_file(p))?, l.dataset.clone()); | ||||||||||||||||
| Some(super::reads::ahead("Source File State", ahead_key(id), move || { | ||||||||||||||||
| let changed = synced.files.changed(std::path::Path::new(&path)).map_err(|e| e.to_string()); | ||||||||||||||||
| let rows = rustgis_data::sync::rows_digest(&dataset).map_err(|e| e.to_string()); | ||||||||||||||||
| Checked { synced, path, dataset, rows, changed } | ||||||||||||||||
| })) | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| #[cfg(target_arch = "wasm32")] | ||||||||||||||||
| pub(crate) fn sync_ahead(_: &Session, _: &Value) -> Option<super::reads::ReadWork> { | ||||||||||||||||
| None | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| /// Compares the layer and the file it was read from, each with the record of them made when it | ||||||||||||||||
| /// was read: whether the layer's rows differ from the rows read (edits the file lacks), and which | ||||||||||||||||
| /// parts of the file differ from the parts read. It doesn't compare the layer with the file as | ||||||||||||||||
| /// it is now, so the same edit made to both shows as `both`. Each part of the size recorded is | ||||||||||||||||
| /// read and hashed, and the rows are hashed, so the work grows with the file. The desktop app | ||||||||||||||||
| /// does it on a thread of its own first ([`sync_ahead`]), with progress and **Cancel**, and the | ||||||||||||||||
| /// command takes that check as it lands; when the layer changed meanwhile it says so rather than | ||||||||||||||||
| /// check again here, which would hold the window. The assistant, scripts and the command line | ||||||||||||||||
| /// check where they ask. | ||||||||||||||||
| fn sync_state(s: &mut Session, p: &Value) -> CmdResult { | ||||||||||||||||
| let id = s.layer_id(p)?; | ||||||||||||||||
| let l = s.map().layer(id).ok_or("the layer no longer exists")?; | ||||||||||||||||
| let (Some(synced), Some(path)) = (&l.synced, &l.dataset.source) else { | ||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Security (low-medium confidence): the check trusts the project's
On Windows, a UNC path such as Suggested fix: refuse the check unless
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 6aa5de8. |
||||||||||||||||
| if l.dataset.source.as_deref().is_some_and(rustgis_data::sync::is_syncable) && !l.is_streamed() { | ||||||||||||||||
| return Err(format!( | ||||||||||||||||
| "{} has no record of its file as it was read (it was added before RustGIS kept one, a part of the file couldn't be read, or an older RustGIS saved the project): add the file again to compare it", | ||||||||||||||||
| l.name | ||||||||||||||||
| )); | ||||||||||||||||
| } | ||||||||||||||||
| return Err(format!("{} wasn't read whole from a Shapefile or .geojson file, so there's no file to compare it with", l.name)); | ||||||||||||||||
| }; | ||||||||||||||||
| // A project from elsewhere names the path: only a local file of a kind a record is made for | ||||||||||||||||
| // is opened, never a network share (opening one can send the user's login to its host). | ||||||||||||||||
| if !local_file(path) { | ||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bug (medium-low confidence): a relative Two fixes, and the first is preferable: store
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 0230eef. When the record is made, |
||||||||||||||||
| return Err(format!("{}: {path} isn't a Shapefile or .geojson file on this computer, so it isn't checked", l.name)); | ||||||||||||||||
| } | ||||||||||||||||
| #[cfg(target_arch = "wasm32")] | ||||||||||||||||
| { | ||||||||||||||||
| let _ = (synced, path); | ||||||||||||||||
| Err("the web app can't read files on this computer".into()) | ||||||||||||||||
| } | ||||||||||||||||
| #[cfg(not(target_arch = "wasm32"))] | ||||||||||||||||
| { | ||||||||||||||||
| let (was, files, path) = (synced.rows, synced.files.clone(), path.clone()); | ||||||||||||||||
| let ahead = super::reads::take::<Checked>(s, &ahead_key(id)); | ||||||||||||||||
| let l = s.map().layer(id).ok_or("the layer no longer exists")?; | ||||||||||||||||
| let (rows, changed) = match ahead { | ||||||||||||||||
| Some(checked) if checked.of(l) => (checked.rows?, checked.changed?), | ||||||||||||||||
| // Checking again here would hold the window: the check made on its thread is dropped. | ||||||||||||||||
| Some(_) => return Err("the layer or its file's record changed while it was checked: check again".into()), | ||||||||||||||||
| None => ( | ||||||||||||||||
| rustgis_data::sync::rows_digest(&l.dataset).map_err(|e| e.to_string())?, | ||||||||||||||||
| files.changed(std::path::Path::new(&path)).map_err(|e| e.to_string())?, | ||||||||||||||||
| ), | ||||||||||||||||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||||||||||||||||
| }; | ||||||||||||||||
| let edited = rows != was; | ||||||||||||||||
| let state = match (edited, !changed.is_empty()) { | ||||||||||||||||
| (false, false) => "inSync", | ||||||||||||||||
| (true, false) => "edited", | ||||||||||||||||
| (false, true) => "sourceChanged", | ||||||||||||||||
| (true, true) => "both", | ||||||||||||||||
| }; | ||||||||||||||||
| Ok(serde_json::json!({"layer": id, "path": path, "state": state, "edited": edited, "changed": changed})) | ||||||||||||||||
| } | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| /// Whether `path` names a file on this computer a record is made for: a Shapefile or `.geojson` | ||||||||||||||||
| /// file, not an address, a database or a network share (`\\host\share`, `//host/share`, | ||||||||||||||||
| /// `/\host\share`, `\\?\UNC\host\share`). Windows' long form of a local path (`\\?\C:\…`) is one | ||||||||||||||||
| /// on this computer. | ||||||||||||||||
| fn local_file(path: &str) -> bool { | ||||||||||||||||
| let b = rustgis_map::paths::local_path(path).as_bytes(); | ||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Security (low, medium confidence): the guard checks one string and the open uses another. Add Data never writes a
Suggested change
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agreed that the checked and opened strings should be the same, and refusing the |
||||||||||||||||
| let sep = |i: usize| b.get(i).is_some_and(|c| matches!(c, b'/' | b'\\')); | ||||||||||||||||
| // Windows takes any two leading separators, of either kind, for a network share, unless they | ||||||||||||||||
| // begin the long form of a path on a drive. | ||||||||||||||||
| let drive = b.get(2) == Some(&b'?') && sep(3) && b.get(4).is_some_and(u8::is_ascii_alphabetic) && b.get(5) == Some(&b':') && sep(6); | ||||||||||||||||
| let share = sep(0) && sep(1) && !drive; | ||||||||||||||||
| rustgis_map::paths::is_local(path) && rustgis_data::sync::is_syncable(path) && !share | ||||||||||||||||
| } | ||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Performance (medium-high confidence): a file over the reader's cap is hashed in full before it is refused.
add_data_aheadtakes a.geojsonwith"stream": falseat any size (m.len() <= IN_PLACE_ABOVEonly applies whenstreamis absent).read_wholethen callsread_synced, which runsFileSync::aroundand hashes every part beforeread_path'scheck_sizerefuses the file aboveMAX_BYTES(256 MiB). A 4 GB GeoJSON therefore spends about 16 s hashing (at ~250 MB/s) before the same error that used to come back at once. The.shppath has the same ordering.The delimited branch just above already checks
MAX_BYTESbefore reading. The record only makes sense for a file the reader will take, so the check belongs here:An oversized file then goes to
read()and fails with the usual "file is too large" error, with no hashing and no record. A test that a"stream": falsefile over the cap is refused fast would pin this.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 203ced5, in
read_syncedso both callers get it: a file pastMAX_BYTES, or a Shapefile whose.dbfis, goes straight to the read, which refuses it as before, with no hashing and no record. I left out the test: it would need a file over 256 MiB, and on the Windows runners that may actually be allocated.