Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions crates/data/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ tokio-util = { workspace = true }
rustls = { workspace = true }
webpki-roots = { workspace = true }
# Native web clients verify servers with the system's trust (see `http`).
# Hashes a source's files, to tell whether they changed since a layer was read (`sync`).
sha2 = { workspace = true }
ureq = { version = "3", default-features = false, features = ["rustls", "gzip", "platform-verifier"] }
# Private temporary workspaces for downloaded JPEG 2000 conversion through GDAL.
tempfile = "3"
Expand Down
8 changes: 5 additions & 3 deletions crates/data/src/dbf.rs
Original file line number Diff line number Diff line change
Expand Up @@ -181,11 +181,13 @@ pub(crate) fn read_cpg(path: &std::path::Path, notes: &mut Vec<String>) -> Optio
}
}

/// Every ASCII case of the `.cpg` extension, the conventional spellings first: tried in turn,
/// without scanning a potentially large directory, and the first there wins.
pub(crate) const CPG_SPELLINGS: [&str; 8] = ["cpg", "CPG", "Cpg", "cPg", "cpG", "CPg", "CpG", "cPG"];

fn try_read_cpg(path: &std::path::Path) -> std::io::Result<Option<Vec<u8>>> {
use std::io::Read;
// Check every ASCII case combination without scanning a potentially large directory.
// Prefer the conventional spellings if multiple sidecars exist.
for ext in ["cpg", "CPG", "Cpg", "cPg", "cpG", "CPg", "CpG", "cPG"] {
for ext in CPG_SPELLINGS {
match std::fs::File::open(path.with_extension(ext)) {
Ok(file) => {
let mut bytes = Vec::new();
Expand Down
5 changes: 5 additions & 0 deletions crates/data/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ pub mod sheets;
pub mod source;
pub mod source_layers;
pub mod sqlite;
pub mod sync;
pub mod threads;
pub mod time;
pub mod topojson;
Expand Down Expand Up @@ -377,6 +378,10 @@ impl Serialize for Dataset {
pub struct ReadReport {
pub skipped: usize,
pub messages: Vec<String>,
/// The files read and the rows read from them, when the read recorded them
/// ([`sync::FileSync::around`]): what a layer kept in the project compares with its source
/// before writing edits back.
pub synced: Option<sync::Synced>,
}

/// What a read says when it drops Z or M values (see [`Dataset::zm_dropped`]): from `features` of
Expand Down
1 change: 1 addition & 0 deletions crates/data/src/source/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -562,6 +562,7 @@ fn materialize_within(src: &dyn FeatureSource, budget: u64) -> Result<(Dataset,
let report = ReadReport {
skipped: skipped + src.skipped_at_open(),
messages: info.notes.iter().map(|n| format!("{}: {n}", info.name)).collect(),
synced: None,
};
if ds.geometry_type == GeometryType::None
&& let Some(g) = ds.features.iter().find_map(|f| f.geometry.as_ref())
Expand Down
347 changes: 347 additions & 0 deletions crates/data/src/sync.rs

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions crates/engine/src/cmd/items.rs
Original file line number Diff line number Diff line change
Expand Up @@ -659,6 +659,7 @@ fn template(p: &Project) -> (Project, TemplateLosses) {
// A feature-less template has no loaded rows to save back to a source.
l.postgis = None;
l.gpkg = None;
l.synced = None;
if let Some(rl) = &mut l.raster {
rl.embedded = None;
}
Expand Down
109 changes: 96 additions & 13 deletions crates/engine/src/cmd/layer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1401,8 +1401,11 @@ fn add_data(s: &mut Session, p: &Value) -> CmdResult {
let src = rustgis_data::source::shp::ShpSource::open_path_in(Path::new(path), name, Some(crs)).map_err(|e| e.to_string())?;
return add_source_with(s, p, Origin::Path(path), Some(Box::new(src)), None);
}
let mut read = rustgis_data::io::read_shapefile_in(Path::new(path), name, Some(crs)).map_err(|e| e.to_string())?;
read.0.source = Some(path.to_string());
// Read ahead in the desktop app ([`add_data_ahead`]).
let read = match super::reads::take(s, &whole_key(path)) {
Some(read) => read,
None => read_shapefile_with(path, name, crs),
}?;
return add_read(s, p, read, None);
}
// A large Shapefile is read in place, as a GeoPackage is; smaller ones are loaded whole.
Expand Down Expand Up @@ -1580,19 +1583,72 @@ fn read_whole(p: &Value, path: &str) -> WholeRead {
}
std::fs::read(path).map_err(|e| format!("{path}: {e}"))
})?;
match delimited {
Some(read) => Ok(read),
None => rustgis_data::io::read_path(Path::new(path)).map_err(|e| format!("{path}: {e}")),
if let Some(read) = delimited {
return Ok(read);
}
let read = || rustgis_data::io::read_path(Path::new(path)).map_err(|e| format!("{path}: {e}"));
// A file its edits can be written back to: the files as read, to compare with before then.
#[cfg(not(target_arch = "wasm32"))]
if rustgis_data::sync::is_syncable(path) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance (medium-high confidence): a file over the reader's cap is hashed in full before it is refused.

add_data_ahead takes a .geojson with "stream": false at any size (m.len() <= IN_PLACE_ABOVE only applies when stream is absent). read_whole then calls read_synced, which runs FileSync::around and hashes every part before read_path's check_size refuses the file above MAX_BYTES (256 MiB). A 4 GB GeoJSON therefore spends about 16 s hashing (at ~250 MB/s) before the same error that used to come back at once. The .shp path has the same ordering.

The delimited branch just above already checks MAX_BYTES before reading. The record only makes sense for a file the reader will take, so the check belongs here:

Suggested change
if rustgis_data::sync::is_syncable(path) {
if rustgis_data::sync::is_syncable(path) && std::fs::metadata(path).is_ok_and(|m| m.len() <= rustgis_data::io::MAX_BYTES) {
return read_synced(path, read);
}

An oversized file then goes to read() and fails with the usual "file is too large" error, with no hashing and no record. A test that a "stream": false file over the cap is refused fast would pin this.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 203ced5, in read_synced so both callers get it: a file past MAX_BYTES, or a Shapefile whose .dbf is, goes straight to the read, which refuses it as before, with no hashing and no record. I left out the test: it would need a file over 256 MiB, and on the Windows runners that may actually be allocated.

return read_synced(path, read);
}
read()
}

/// A Shapefile read whole in coordinate system `crs`, which replaces its `.prj`, with the record
/// of what was read ([`read_synced`]).
#[cfg(not(target_arch = "wasm32"))]
fn read_shapefile_with(path: &str, name: &str, crs: rustgis_data::raster::Crs) -> WholeRead {
read_synced(path, || {
let mut read = rustgis_data::io::read_shapefile_in(Path::new(path), name, Some(crs)).map_err(|e| e.to_string())?;
read.0.source = Some(path.to_string());
Ok(read)
})
}

/// `read`, the read of the file at `path`, with the record of what it read: the files hashed
/// just before ([`rustgis_data::sync::FileSync::around`]) and the digest of the rows read. Its
/// work grows with the file, so the desktop app makes it off the UI thread ([`add_data_ahead`]).
/// When the files couldn't be hashed the layer has no record, and a note says it can't be
/// compared with its file later.
#[cfg(not(target_arch = "wasm32"))]
fn read_synced(path: &str, read: impl FnOnce() -> WholeRead) -> WholeRead {
// A file past what the readers take (a Shapefile's `.dbf` too) is refused by the read: not
// hashed first.
let over = |p: &Path| std::fs::metadata(p).is_ok_and(|m| m.len() > rustgis_data::io::MAX_BYTES);
let shp = path.to_ascii_lowercase().ends_with(".shp");
if over(Path::new(path)) || (shp && over(&Path::new(path).with_extension("dbf"))) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test coverage (AGENTS.md, Test both ways): the over-cap early return has no test. c06cb93 and 203ced5 changed this branch, and nothing in the repo exercises it. I grepped crates/ for MAX_BYTES and set_len in test files and found no case that makes a file past the cap.

Two cases need a regression test:

  • a .geojson with a sibling .dbf past MAX_BYTES must still get a record (synced.is_some(), state inSync). Before c06cb93, the .dbf check ran for every path, so this file got no record and no notice.
  • a .shp whose .dbf is past the cap must be refused by the read, with no record and no hashing.

A sparse file keeps this cheap: set_len(rustgis_data::io::MAX_BYTES + 1) only writes metadata on Linux and macOS, and the check reads only metadata. Gate the test on #[cfg(unix)] so Windows doesn't allocate 256 MiB.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this without a test, as in the earlier thread on this branch. Exercising it needs a file over MAX_BYTES (256 MiB), and on the Windows runners (ci:windows) that may really be allocated rather than sparse. The branch only returns to the existing read, which refuses the file with its usual error. A test with a smaller cap passed in is left for the release quality pass.

return read();
}
let (read, files) = rustgis_data::sync::FileSync::around(Path::new(path), read).map_err(|e| format!("{path}: {e}"))?;
let (mut ds, mut report) = read?;
// Kept whole, as the record checks it later: a relative path would be taken from wherever
// RustGIS runs then, not where the file was read.
if files.is_some()
&& let Ok(full) = std::path::absolute(path)
{
ds.source = Some(full.display().to_string());
}
report.synced = match files {
Some(files) => {
Some(rustgis_data::sync::Synced { files, rows: rustgis_data::sync::rows_digest(&ds).map_err(|e| format!("{path}: {e}"))? })
}
None => {
report.messages.push(format!(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AGENTS.md (Never silent, test both ways): the notice on a part that can't be hashed has no test. The "couldn't be read to record it" branch of read_synced is not covered by any test. Only clean reads are. Add one that gives a lossy read and asserts the notice and synced == None, plus a clean read that asserts no such notice. One way to make the failure, if the reader accepts it: make the .prj beside a Shapefile a directory, so FileSync::read fails on it while the read still succeeds.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 0230eef. The engine test adds a Shapefile with a folder where its .prj would be. The read succeeds, the layer has no record, and the "couldn't be read to record it" notice is given once. A clean read of the GeoJSON gives no such notice.

"{path}: a part of it couldn't be read to record it, so RustGIS can't tell later whether the file changed since; add it again to compare it"
));
None
}
};
Ok((ds, report))
}

/// Add Data's read of the file `p` names, made before it runs ([`super::reads`]): a raster read
/// whole or converted by GDAL, a GeoJSON read whole, delimited text, a spreadsheet or a city model
/// (CityJSON or CityGML) read into its layers. `None` for
/// (CityJSON or CityGML) read into its layers, a Shapefile read whole. `None` for
/// any other file, which [`add_data`] reads as it adds it. Each is picked only where `add_data`
/// takes that read, so a read made is never wasted; a few it makes whole are left to it (a GeoJSON
/// with `"stream"` given, a `.json` with a coordinate system or layer given, a large GeoJSON that
/// isn't a FeatureCollection).
/// takes that read, so a read made is never wasted; a few it makes whole are left to it (a `.json`
/// with a coordinate system or layer given, a large GeoJSON that isn't a FeatureCollection).
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn add_data_ahead(p: &Value) -> Option<super::reads::ReadWork> {
use super::reads::ahead;
Expand All @@ -1616,19 +1672,42 @@ pub(crate) fn add_data_ahead(p: &Value) -> Option<super::reads::ReadWork> {
}
let given = |k: &str| ps(p, k).is_some_and(|v| !v.trim().is_empty());
let lower = path.to_ascii_lowercase();
// A GeoJSON read whole: small, with no "stream" given (which add_data decides by), and not a
// CityJSON model (named so, or a .json with a coordinate system or a layer given).
// A GeoJSON read whole: small with no "stream" given (which add_data decides by), or any size
// (within the reader's limit) with `"stream": false`; and not a CityJSON model (named so, or a
// .json with a coordinate system or a layer given).
let geojson = is_geojson(&path)
&& !rustgis_data::cityjson::is_cityjson_name(&path)
&& b(p, "stream").is_none()
&& b(p, "stream") != Some(true)
&& !(lower.ends_with(".json") && (given("crs") || given("layer")))
&& std::fs::metadata(&path).is_ok_and(|m| m.is_file() && m.len() <= rustgis_data::source::geojson::IN_PLACE_ABOVE);
&& std::fs::metadata(&path)
.is_ok_and(|m| m.is_file() && (b(p, "stream") == Some(false) || m.len() <= rustgis_data::source::geojson::IN_PLACE_ABOVE));
// A city model: its layers, read whole. A model named `.json` (not `.city.json`) is a file of
// several layers only with a coordinate system or a layer given, which Add Data reads as it runs.
if rustgis_data::io::is_multi_layer_name(&path) && is_city_model_path(&path) {
let (key, q) = (city_key(&path), p.clone());
return Some(ahead(NAME, key, move || read_city_file(&q, &path)));
}
// A Shapefile read whole, in the coordinate system given if one is: as add_data reads it (one
// past the whole-file reader's cap, or with more records than it takes, is read in place).
if lower.ends_with(".shp") {
let q = past_cap_shapefile(&path, p).unwrap_or_else(|| p.clone());
let stream = b(&q, "stream").unwrap_or_else(|| {
rustgis_data::source::shp::record_count(Path::new(&path)).is_some_and(|n| n > rustgis_map::source::STREAM_ABOVE)
});
if stream {
return None;
}
let key = whole_key(&path);
return match ps(&q, "crs").map(str::trim).filter(|c| !c.is_empty()) {
Some(text) => {
// One add_data can't parse it refuses as it runs.
let crs = rustgis_data::raster::Crs::parse(text).ok()?;
let name = Path::new(&path).file_stem().and_then(|n| n.to_str()).unwrap_or("layer").to_string();
Some(ahead(NAME, key, move || read_shapefile_with(&path, &name, crs)))
}
None => Some(ahead(NAME, key, move || read_whole(&q, &path))),
};
}
let (key, q) = (whole_key(&path), p.clone());
if rustgis_data::sheets::is_sheet_name(&path) {
return Some(ahead(NAME, key, move || read_sheet_file(&q, &path)));
Expand Down Expand Up @@ -4731,7 +4810,11 @@ pub(crate) fn add_read(
}
let n = ds.features.len();
let empty_before = super::map_is_empty(s);
let synced = report.synced;
let id = add_dataset(s, ds);
if let Some(l) = s.map_mut().layer_mut(id) {
l.synced = synced;
}
if zoom.unwrap_or(empty_before)
&& let Some(e) = s.map().layer(id).map(|l| l.extent())
{
Expand Down
3 changes: 3 additions & 0 deletions crates/engine/src/cmd/layer_copy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ fn copy(s: &mut Session, p: &Value) -> CmdResult {
.map(|l| {
let mut l = l.clone();
l.selection.clear();
// A copy isn't the layer read from its file: it can't say whether the file has its
// edits, nor write them back.
l.synced = None;
l
})
.collect();
Expand Down
1 change: 1 addition & 0 deletions crates/engine/src/cmd/layout.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1686,6 +1686,7 @@ fn publication(s: &mut Session, p: &Value) -> CmdResult {
// Simplified sketches aren't the source's rows: saving them would overwrite it.
layer.gpkg = None;
layer.postgis = None;
layer.synced = None;
layer.dataset.features = l
.dataset
.features
Expand Down
2 changes: 2 additions & 0 deletions crates/engine/src/cmd/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ pub mod plugins;
pub mod portals;
pub mod presentation;
pub mod project;
pub mod publish;
pub mod qgis;
pub mod raster;
pub mod reads;
Expand Down Expand Up @@ -112,6 +113,7 @@ pub fn all() -> Vec<CommandSpec> {
v.extend(sql::commands());
v.extend(pg::commands());
v.extend(gpkg::commands());
v.extend(publish::commands());
v.extend(plugins::commands());
v.extend(sharing::commands());
v.extend(stac::commands());
Expand Down
133 changes: 133 additions & 0 deletions crates/engine/src/cmd/publish.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
//! A layer kept in the project and the file it was read from: whether its edits and the file are
//! in step (#1385).

use crate::{CmdResult, CommandSpec as C, Session};
use serde_json::Value;

pub fn commands() -> Vec<C> {
vec![
C::new("source.syncState", "Source File State", "Data › Data Source", sync_state).roles(&[]).effects(crate::policy::Effects::NONE.with(crate::policy::Effect::Files)).view().params(
r#"{"layer"?: "name|id (default: the active layer): one read whole from a Shapefile or .geojson file"} — whether it has edits its file lacks ("edited"), the file changed since it was read ("sourceChanged"), both ("both") or neither ("inSync"), and which of the file's parts changed"#,
),
]
}

/// The key a check of layer `id`'s files made ahead is taken by (see [`super::reads`]).
#[cfg(not(target_arch = "wasm32"))]
fn ahead_key(id: u32) -> String {
format!("syncState:{id}")
}

/// A check of a layer made ahead: the record, path and rows checked (the rows shared with the
/// layer, not copied), the rows' digest and what changed in the files.
#[cfg(not(target_arch = "wasm32"))]
struct Checked {
synced: rustgis_data::sync::Synced,
path: String,
dataset: rustgis_data::Dataset,
rows: Result<u64, String>,
changed: Result<Vec<String>, String>,
}

#[cfg(not(target_arch = "wasm32"))]
impl Checked {
/// Whether it checked layer `l` as it is: the same record and path, and the same rows (the
/// same allocation: an edit copies them), fields and coordinate system.
fn of(&self, l: &rustgis_map::Layer) -> bool {
l.synced.as_ref() == Some(&self.synced)
&& l.dataset.source.as_deref() == Some(self.path.as_str())
&& l.dataset.features.shares(&self.dataset.features)
&& l.dataset.fields == self.dataset.fields
&& l.dataset.crs == self.dataset.crs
}
}

/// The check `source.syncState` makes before it runs in the desktop app: every part of the
/// file read and hashed, and the layer's rows hashed, on a thread of its own with progress and
/// Cancel.
#[cfg(not(target_arch = "wasm32"))]
pub(crate) fn sync_ahead(s: &Session, p: &Value) -> Option<super::reads::ReadWork> {
let id = s.layer_id(p).ok()?;
let l = s.map().layer(id)?;
let (synced, path, dataset) = (l.synced.clone()?, l.dataset.source.clone().filter(|p| local_file(p))?, l.dataset.clone());
Some(super::reads::ahead("Source File State", ahead_key(id), move || {
let changed = synced.files.changed(std::path::Path::new(&path)).map_err(|e| e.to_string());
let rows = rustgis_data::sync::rows_digest(&dataset).map_err(|e| e.to_string());
Checked { synced, path, dataset, rows, changed }
}))
}

#[cfg(target_arch = "wasm32")]
pub(crate) fn sync_ahead(_: &Session, _: &Value) -> Option<super::reads::ReadWork> {
None
}

/// Compares the layer and the file it was read from, each with the record of them made when it
/// was read: whether the layer's rows differ from the rows read (edits the file lacks), and which
/// parts of the file differ from the parts read. It doesn't compare the layer with the file as
/// it is now, so the same edit made to both shows as `both`. Each part of the size recorded is
/// read and hashed, and the rows are hashed, so the work grows with the file. The desktop app
/// does it on a thread of its own first ([`sync_ahead`]), with progress and **Cancel**, and the
/// command takes that check as it lands; when the layer changed meanwhile it says so rather than
/// check again here, which would hold the window. The assistant, scripts and the command line
/// check where they ask.
fn sync_state(s: &mut Session, p: &Value) -> CmdResult {
let id = s.layer_id(p)?;
let l = s.map().layer(id).ok_or("the layer no longer exists")?;
let (Some(synced), Some(path)) = (&l.synced, &l.dataset.source) else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security (low-medium confidence): the check trusts the project's dataset.source path without the local-path check that Reload Data uses.

reload_input (crates/engine/src/cmd/repair.rs:147-152) requires rustgis_map::paths::is_local(path). This path has no such check. The only guard is the extension, which is_syncable applies at read time, not when a saved project is opened. A project from elsewhere can set dataset.source and a matching synced record. Running source.syncState on that layer then calls metadata and File::open on whatever path it names.

On Windows, a UNC path such as \\host\share\x.shp makes the open start an SMB connection, which is a known way to leak credentials. The changed-part names in the output are also an oracle for whether a file's contents match a guessed hash.

Suggested fix: refuse the check unless rustgis_map::paths::is_local(path) && rustgis_data::sync::is_syncable(path) holds, with a test that a \\-style path and a .json path are refused without touching the disk.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6aa5de8. source.syncState and its read-ahead now refuse, without touching the disk, any path that isn't a Shapefile or .geojson file on this computer. That means paths::is_local and sync::is_syncable, plus an explicit refusal of \\\\host\\share and //host/share, since is_local lets UNC paths through. The test covers both share forms and a .json path.

if l.dataset.source.as_deref().is_some_and(rustgis_data::sync::is_syncable) && !l.is_streamed() {
return Err(format!(
"{} has no record of its file as it was read (it was added before RustGIS kept one, a part of the file couldn't be read, or an older RustGIS saved the project): add the file again to compare it",
l.name
));
}
return Err(format!("{} wasn't read whole from a Shapefile or .geojson file, so there's no file to compare it with", l.name));
};
// A project from elsewhere names the path: only a local file of a kind a record is made for
// is opened, never a network share (opening one can send the user's login to its host).
if !local_file(path) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (medium-low confidence): a relative dataset.source is hashed against the process's working directory, not the project's folder. A Shapefile or GeoJSON layer added with a relative path (from the CLI, a script or the assistant) keeps that path as dataset.source. I found no step that absolutises it on this path. Map::file_paths_mut doesn't list dataset.source, so opening the project doesn't resolve it either. If the project is saved, then reopened from another directory, changed() looks at the wrong files or none. Every part then shows as changed and the state reads sourceChanged with no edit to the file. Nothing is lost, but the report is false, and the next PR's publish would refuse.

Two fixes, and the first is preferable: store std::path::absolute(path) as dataset.source in read_shapefile_with and read_whole, or refuse non-absolute paths here. If you refuse them, use is_absolute_local together with the separator check from the comment above, since is_absolute_local has the same gap. I haven't traced the CLI path handling, so please confirm the scenario.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0230eef. When the record is made, read_synced stores the file's absolute path as dataset.source. Saving a project still writes it relative to the project's folder, and opening restores it (the project.saveAs/project.open round trip in the test covers that). So a later check no longer depends on the directory RustGIS runs in.

return Err(format!("{}: {path} isn't a Shapefile or .geojson file on this computer, so it isn't checked", l.name));
}
#[cfg(target_arch = "wasm32")]
{
let _ = (synced, path);
Err("the web app can't read files on this computer".into())
}
#[cfg(not(target_arch = "wasm32"))]
{
let (was, files, path) = (synced.rows, synced.files.clone(), path.clone());
let ahead = super::reads::take::<Checked>(s, &ahead_key(id));
let l = s.map().layer(id).ok_or("the layer no longer exists")?;
let (rows, changed) = match ahead {
Some(checked) if checked.of(l) => (checked.rows?, checked.changed?),
// Checking again here would hold the window: the check made on its thread is dropped.
Some(_) => return Err("the layer or its file's record changed while it was checked: check again".into()),
None => (
rustgis_data::sync::rows_digest(&l.dataset).map_err(|e| e.to_string())?,
files.changed(std::path::Path::new(&path)).map_err(|e| e.to_string())?,
),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
};
let edited = rows != was;
let state = match (edited, !changed.is_empty()) {
(false, false) => "inSync",
(true, false) => "edited",
(false, true) => "sourceChanged",
(true, true) => "both",
};
Ok(serde_json::json!({"layer": id, "path": path, "state": state, "edited": edited, "changed": changed}))
}
}

/// Whether `path` names a file on this computer a record is made for: a Shapefile or `.geojson`
/// file, not an address, a database or a network share (`\\host\share`, `//host/share`,
/// `/\host\share`, `\\?\UNC\host\share`). Windows' long form of a local path (`\\?\C:\…`) is one
/// on this computer.
fn local_file(path: &str) -> bool {
let b = rustgis_map::paths::local_path(path).as_bytes();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security (low, medium confidence): the guard checks one string and the open uses another. local_file normalises the path with local_path (it strips file://), but sync_state and sync_ahead pass the raw dataset.source to files.changed(...). A path such as file:///C:/data/towns.shp passes the guard, then opens the literal file:///… string. That is not a UNC path, so no share is contacted. On Windows, though, every recorded part reads as changed, which gives a false sourceChanged.

Add Data never writes a file:// dataset.source, so this needs a hand-edited project. The fix is to refuse the address form, so the path checked is the path opened:

Suggested change
let b = rustgis_map::paths::local_path(path).as_bytes();
fn local_file(path: &str) -> bool {
// A `file://` address is refused: the path checked is then the path opened.
if rustgis_map::paths::local_path(path) != path {
return false;
}
let b = path.as_bytes();

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that the checked and opened strings should be the same, and refusing the file:// form is the simple way to do it. As you note, no share is reached: only a hand-edited project can hold such a source, and the worst case is a false sourceChanged, which errs on the safe side. Left for the release quality pass.

let sep = |i: usize| b.get(i).is_some_and(|c| matches!(c, b'/' | b'\\'));
// Windows takes any two leading separators, of either kind, for a network share, unless they
// begin the long form of a path on a drive.
let drive = b.get(2) == Some(&b'?') && sep(3) && b.get(4).is_some_and(u8::is_ascii_alphabetic) && b.get(5) == Some(&b':') && sep(6);
let share = sep(0) && sep(1) && !drive;
rustgis_map::paths::is_local(path) && rustgis_data::sync::is_syncable(path) && !share
}
Loading
Loading