diff --git a/.config/nextest.toml b/.config/nextest.toml index ffc1d9fbfe..c14a204e08 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -13,6 +13,6 @@ retries = 2 fail-fast = false [profile.ci.junit] -# target/nextest/ci/junit.xml: the report flaky-tests.sh reads (`` marks a test that -# passed only on a retry). +# target/nextest/ci/junit.xml: the report flaky-tests.sh reads (`` or `` +# marks a test that passed only on a retry). path = "junit.xml" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index efe4bdcba6..b363f299fd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -255,13 +255,15 @@ jobs: RUN: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | set -euo pipefail - [ -d flaky ] || exit 0 - sort -u flaky/flaky-*.txt | while read -r test; do + shopt -s nullglob + files=(flaky/flaky-*.txt) + [ "${#files[@]}" -gt 0 ] || exit 0 + sort -u "${files[@]}" | while read -r test; do title="Flaky test: $test" runners="$(grep -lxF -- "$test" flaky/flaky-*.txt | sed 's|^flaky/flaky-||; s|\.txt$||' | paste -sd, - | sed 's/,/, /g')" body="$test failed and passed on a retry on $runners, for $GITHUB_SHA: $RUN" - open="$(gh issue list -R "$GITHUB_REPOSITORY" --state open --search "in:title \"$title\"" --json number,title \ - | jq -r --arg t "$title" '[.[] | select(.title == $t)][0].number // ""')" + open="$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues?state=open&per_page=100" \ + | jq -sr --arg t "$title" '[.[][] | select(.pull_request | not) | select(.title == $t)][0].number // ""')" if [ -n "$open" ]; then gh issue comment "$open" -R "$GITHUB_REPOSITORY" --body "Flaky again: $body" else diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index be3d44ccb4..b7f1286ad4 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -145,8 +145,8 @@ jobs: run: | set -euo pipefail title="Nightly build failed" - open="$(gh issue list -R "$GITHUB_REPOSITORY" --state open --search "in:title \"$title\"" \ - --json number,title --jq "[.[] | select(.title == \"$title\")][0].number // \"\"")" + open="$(gh api --paginate "repos/$GITHUB_REPOSITORY/issues?state=open&per_page=100" \ + | jq -sr --arg t "$title" '[.[][] | select(.pull_request | not) | select(.title == $t)][0].number // ""')" if [ "$RESULT" = incomplete ]; then echo "The real-data tests didn't run (download failed): leaving the nightly issue as it is." elif [ "$RESULT" = success ]; then diff --git a/.github/workflows/test-rust.yml b/.github/workflows/test-rust.yml index f1336832d8..96f4089b4d 100644 --- a/.github/workflows/test-rust.yml +++ b/.github/workflows/test-rust.yml @@ -72,6 +72,9 @@ jobs: - uses: taiki-e/install-action@cfdb7ed4ccc7c7dce37759eae89dc5674b469b39 # v2 with: tool: cargo-nextest@0.9.151 + - name: Clear the previous test report + shell: bash + run: rm -f target/nextest/ci/junit.xml # PACKAGES is several words, so it is left unquoted. A pull request that changes only a # package with no tests runs none (`--no-tests=warn`). - name: Tests diff --git a/AGENTS.md b/AGENTS.md index 73bbed7462..b18fff3a61 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,8 @@ is out of scope. - A new background raster command or export, or a tool whose work grows with its input, goes in the lists in `crates/engine/src/tests.rs` (`cancel_rasters`, `cancel_exports`, `cancel_tools`); `cancel_rasters` and `cancel_exports` must match the background registries. A read-ahead command - (`reads::prepare`, such as `service.layers`) has no list: add its call to the test below by hand. + (`reads::prepare`, such as `service.layers`) has no list: add its call to + `long_commands_stop_when_cancelled_while_they_run` by hand. `long_commands_stop_when_cancelled_while_they_run` fails one that never looks at Cancel; `cancelling_at_each_check_leaves_nothing_behind` cancels each command of the three lists at its first three checks and its last, and fails one that leaves the project, its undo steps or the @@ -108,7 +109,7 @@ Work that splits into independent parts runs on several threads where practical: A change that works where you tried it is often missing elsewhere. Before calling it done, say in the pull request which of these it reaches and why the rest don't apply: - the desktop app, the browser (file access through `Services`), the command line, Python, the - phone (the touch bar); + phone (the touch interface, `ui-touch`); - the ribbon, shortcuts and command search; the Assistant and MCP (do their tool descriptions show the new option?); plugins; - the way back: what you add can be undone, removed or reopened. @@ -137,7 +138,8 @@ packages ship (the release build fails on stale ones). Pull requests run CI on Linux (plus the Android and iOS builds when they touch `apps/rustgis-mobile`); pushes to `main` test every platform. When a pull request touches platform-specific code, paths or file dialogs, label it `ci:windows` or `ci:macos` to test there, -or `ci:full` for every platform and the release build. +or `ci:full` for every platform and the release build. Label-triggered runs start the tests +immediately; label-gated build jobs in `ci.yml` start from the next push. ## Review comments - Fix in the pull request what a review finds wrong: bugs, crash paths, silent data loss, a broken diff --git a/ROADMAP.md b/ROADMAP.md index 5df0619a3f..17d68da6f2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -285,9 +285,7 @@ attributes kept. - A scene with a free camera and each layer's elevation mode (on the ground, relative, absolute), Phase 2 of [#726](https://github.com/opengeos/RustGIS/issues/726), after the Z design above. - LiDAR point layers (LAS/LAZ/COPC, read today as points) drawn at their heights in the scene. -- Terrain from several layers ([#1310](https://github.com/opengeos/RustGIS/issues/1310)), and - finer terrain once a tilted view settles - ([#1523](https://github.com/opengeos/RustGIS/issues/1523)). +- Terrain from several layers ([#1310](https://github.com/opengeos/RustGIS/issues/1310)). - 3D objects and 3D Tiles follow. ## Exploratory diff --git a/apps/rustgis-cli/src/main.rs b/apps/rustgis-cli/src/main.rs index 9f1d36a2ea..08dec47fe5 100644 --- a/apps/rustgis-cli/src/main.rs +++ b/apps/rustgis-cli/src/main.rs @@ -223,12 +223,16 @@ fn choose_provider(args: &[String], env: &Environment, saved: Option<&Saved>) -> /// Builds a request's configuration without reading unrelated credentials during management. /// A prompt may use the desktop's API key for its provider and address; model listing uses /// only environment API keys, so `--models` never prompts for a saved API key. -fn configuration(provider: &str, args: &[String], env: &Environment, saved: Option<&Saved>) -> Config { +fn configuration(provider: &str, args: &[String], env: &Environment, saved: Option<&Saved>) -> Result { let mut cfg = (env.config)(provider); - if cfg.login() - && let Some(c) = rustgis_assistant::chatgpt::load_any() - { - cfg.sign_in(&c); + if cfg.login() { + let (credentials, warning) = rustgis_assistant::chatgpt::load_any_report(); + if let Some(warning) = warning { + return Err(format!("could not read saved ChatGPT sign-ins: {warning}")); + } + if let Some(c) = credentials { + cfg.sign_in(&c); + } } if let Some(m) = opt(args, "--model").map(str::to_string).or_else(|| env.model.clone()) { cfg.model = m; @@ -243,7 +247,7 @@ fn configuration(provider: &str, args: &[String], env: &Environment, saved: Opti { cfg.key = key.expose().to_string(); } - cfg + Ok(cfg) } const NEEDS_REQUEST: &str = "ask needs a request, e.g. rustgis-cli ask --sample \"Which countries have the largest GDP?\""; @@ -374,7 +378,7 @@ fn ask(s: &mut Session, args: &[String], saved: Option<&Saved>, env: &Environmen if !models && prompt.is_none() { return Err(NEEDS_REQUEST.into()); } - let mut cfg = configuration(&provider, args, env, saved); + let mut cfg = configuration(&provider, args, env, saved)?; // A signed-in account's requests renew its tokens as needed. // Without a usable sign-in, say how to sign in before asking for anything. if cfg.login() @@ -1051,14 +1055,14 @@ mod tests { fn a_prompt_uses_the_saved_key_but_model_listing_does_not_read_it() { let store = anthropic_key(); let opted_in = saved(remembered(), &store); - let list = configuration("anthropic", &args(&["ask", "--models"]), &bare(), Some(&opted_in)); + let list = configuration("anthropic", &args(&["ask", "--models"]), &bare(), Some(&opted_in)).unwrap(); let request = provider::models_request(&list); assert!(request.headers.iter().all(|(_, value)| !value.contains("stored-sentinel"))); assert!(store.accounts().is_empty()); - let environment_list = configuration("openai", &args(&["ask", "--models"]), &with_openai_key(), Some(&opted_in)); + let environment_list = configuration("openai", &args(&["ask", "--models"]), &with_openai_key(), Some(&opted_in)).unwrap(); assert_eq!(environment_list.key, "env-key"); assert!(store.accounts().is_empty()); - let prompt = configuration("anthropic", &args(&["ask", "--provider", "anthropic", "hi"]), &bare(), Some(&opted_in)); + let prompt = configuration("anthropic", &args(&["ask", "--provider", "anthropic", "hi"]), &bare(), Some(&opted_in)).unwrap(); assert_eq!(prompt.key, "stored-sentinel"); assert_eq!(store.accounts(), ["assistant/anthropic"]); } @@ -1067,17 +1071,17 @@ mod tests { fn without_the_opt_in_or_consent_no_key_is_read_and_the_environment_key_wins() { let store = anthropic_key(); // No --use-saved-secrets: the model list goes out without the saved key. - let request = provider::models_request(&configuration("anthropic", &args(&["ask", "--models"]), &bare(), None)); + let request = provider::models_request(&configuration("anthropic", &args(&["ask", "--models"]), &bare(), None).unwrap()); assert!(request.headers.iter().all(|(_, value)| !value.contains("stored-sentinel"))); // Opted in, but the desktop does not remember keys. let off = saved(desktop(&[(provider::REMEMBER_KEY_PREF, "off")]), &store); - assert_eq!(configuration("anthropic", &args(&["ask", "hi"]), &bare(), Some(&off)).key, ""); + assert_eq!(configuration("anthropic", &args(&["ask", "hi"]), &bare(), Some(&off)).unwrap().key, ""); // A key from the environment is the one used, and nothing is read. let opted_in = saved(remembered(), &store); - assert_eq!(configuration("openai", &args(&["ask", "hi"]), &with_openai_key(), Some(&opted_in)).key, "env-key"); + assert_eq!(configuration("openai", &args(&["ask", "hi"]), &with_openai_key(), Some(&opted_in)).unwrap().key, "env-key"); // The desktop sends its key to its own address only: another one gets no key. let elsewhere = args(&["ask", "--base-url", "https://elsewhere.example/v1", "hi"]); - assert_eq!(configuration("anthropic", &elsewhere, &bare(), Some(&opted_in)).key, ""); + assert_eq!(configuration("anthropic", &elsewhere, &bare(), Some(&opted_in)).unwrap().key, ""); assert!(store.accounts().is_empty()); } @@ -1103,6 +1107,22 @@ mod tests { false } + /// A refused store is an actionable CLI error, while no store is a clean signed-out state. + #[test] + fn a_refused_chatgpt_store_is_reported_before_a_cli_request() { + if !in_isolated_settings("tests::a_refused_chatgpt_store_is_reported_before_a_cli_request") { + return; + } + let path = rustgis_assistant::chatgpt::auth_path().unwrap(); + let text = "{\"version\":3,\"accounts\":[]}"; + std::fs::write(&path, text).unwrap(); + let error = configuration("chatgpt", &args(&["ask", "hi"]), &bare(), None).err().unwrap(); + assert!(error.contains("could not read saved ChatGPT sign-ins") && error.contains("newer RustGIS"), "{error}"); + assert_eq!(std::fs::read_to_string(&path).unwrap(), text); + std::fs::remove_file(&path).unwrap(); + assert!(configuration("chatgpt", &args(&["ask", "hi"]), &bare(), None).unwrap().key.is_empty()); + } + #[test] fn managing_the_chatgpt_sign_in_never_reads_an_api_key() { if !in_isolated_settings("tests::managing_the_chatgpt_sign_in_never_reads_an_api_key") { diff --git a/apps/rustgis-mobile/Cargo.toml b/apps/rustgis-mobile/Cargo.toml index f0c325e23c..be03cc99bf 100644 --- a/apps/rustgis-mobile/Cargo.toml +++ b/apps/rustgis-mobile/Cargo.toml @@ -30,6 +30,10 @@ too_many_arguments = "allow" type_complexity = "allow" [dev-dependencies] +rustgis-ui-touch = { workspace = true } +eframe = { workspace = true, features = ["default_fonts", "wgpu", "x11", "wayland"] } +serde_json = { workspace = true } +log = { workspace = true } rustgis-engine = { workspace = true } rustgis-ui-egui = { workspace = true } diff --git a/apps/rustgis-mobile/build-apk.sh b/apps/rustgis-mobile/build-apk.sh index 0cabf5db45..0f5dbb2186 100755 --- a/apps/rustgis-mobile/build-apk.sh +++ b/apps/rustgis-mobile/build-apk.sh @@ -96,7 +96,9 @@ strip=$(ls "$ANDROID_NDK_HOME"/toolchains/llvm/prebuilt/*/bin/llvm-strip | head find "$out/apk/lib" -name librustgis_mobile.so -exec "$strip" {} \; # Each segment aligned to 16 KB pages, which Android 15 phones may use and Google Play requires. readelf=$(ls "$ANDROID_NDK_HOME"/toolchains/llvm/prebuilt/*/bin/llvm-readelf | head -1) +checked=0 while IFS= read -r -d '' so; do + checked=$((checked + 1)) # In an assignment of its own, so a readelf that fails stops the script. info=$("$readelf" -lW "$so") for align in $(awk '$1 == "LOAD" { print $NF }' <<< "$info"); do @@ -106,6 +108,7 @@ while IFS= read -r -d '' so; do fi done done < <(find "$out/apk/lib" -name '*.so' -print0) +[ "$checked" -gt 0 ] || { echo "build-apk: no native libraries to check for 16 KB alignment" >&2; exit 1; } # The activity's Java half, compiled without Gradle: javac, then d8 to Android's bytecode. rm -rf "$out/java" && mkdir -p "$out/java/classes" "$out/java/dex" diff --git a/apps/rustgis-mobile/java/org/rustgis/mobile/EventBuffer.java b/apps/rustgis-mobile/java/org/rustgis/mobile/EventBuffer.java new file mode 100644 index 0000000000..739db87e8d --- /dev/null +++ b/apps/rustgis-mobile/java/org/rustgis/mobile/EventBuffer.java @@ -0,0 +1,61 @@ +package org.rustgis.mobile; + +/** Ordered phone events, with bounded position retention and bounded work per Rust frame. */ +final class EventBuffer { + static final int PER_FRAME = 512; + private final java.util.ArrayDeque events = new java.util.ArrayDeque<>(); + private final int mostPositions; + private int positions; + private long dropped; + + EventBuffer(int mostPositions) { + this.mostPositions = mostPositions; + } + + void add(String event) { + boolean position = event.startsWith("position\t"); + if (position && positions >= mostPositions) { + // Only position fixes may be discarded: picker completions and failures must reach + // Rust even when several days' positions filled the queue before them. + java.util.Iterator pending = events.iterator(); + while (pending.hasNext()) { + if (pending.next().startsWith("position\t")) { + pending.remove(); + positions--; + dropped++; + break; + } + } + } + events.add(event.replace('\n', ' ').replace('\r', ' ')); + if (position) { + positions++; + } + } + + String take() { + StringBuilder taken = new StringBuilder(); + int count = 0; + if (dropped > 0) { + taken.append("failed\t").append(dropped) + .append(" positions found while RustGIS was out of sight were dropped: more were waiting than it keeps."); + dropped = 0; + count++; + } + while (count < PER_FRAME && !events.isEmpty()) { + String event = events.removeFirst(); + if (event.startsWith("position\t")) { + positions--; + } + if (count++ > 0) { + taken.append('\n'); + } + taken.append(event); + } + return taken.toString(); + } + + boolean isEmpty() { + return events.isEmpty() && dropped == 0; + } +} diff --git a/apps/rustgis-mobile/java/org/rustgis/mobile/LocationProviders.java b/apps/rustgis-mobile/java/org/rustgis/mobile/LocationProviders.java new file mode 100644 index 0000000000..2ded02b64e --- /dev/null +++ b/apps/rustgis-mobile/java/org/rustgis/mobile/LocationProviders.java @@ -0,0 +1,23 @@ +package org.rustgis.mobile; + +/** Providers successfully registered for the current listener, confined to Android's main thread. */ +final class LocationProviders { + private final java.util.HashSet requested = new java.util.HashSet<>(); + + void add(String provider) { + requested.add(provider); + } + + void clear() { + requested.clear(); + } + + boolean anyEnabled(java.util.function.Predicate enabled) { + for (String provider : requested) { + if (enabled.test(provider)) { + return true; + } + } + return false; + } +} diff --git a/apps/rustgis-mobile/java/org/rustgis/mobile/MainActivity.java b/apps/rustgis-mobile/java/org/rustgis/mobile/MainActivity.java index 00e2407b9f..9f1576ec0c 100644 --- a/apps/rustgis-mobile/java/org/rustgis/mobile/MainActivity.java +++ b/apps/rustgis-mobile/java/org/rustgis/mobile/MainActivity.java @@ -36,13 +36,18 @@ public class MainActivity extends NativeActivity { private static final int PHOTO = 5; /** * The most events kept for the Rust side: two days of a position a second out of sight. The - * Rust side keeps as many positions between frames (FIXES_KEPT), so a batch goes whole into a track. + * Rust side keeps as many positions between frames (FIXES_KEPT), and each frame drains a bounded batch into its track. */ private static final int MOST_EVENTS = 200_000; private static final int COPY_BUFFER = 1 << 16; + /** Shared by every activity/copy worker for this process, including activity recreation. */ + private static final String COPY_OWNER = java.util.UUID.randomUUID().toString(); + + /** Rust installs this token before inspecting unused document copies. */ + public String copyOwner() { return COPY_OWNER; } /** Oldest first; a deque, so the oldest goes at once when MOST_EVENTS wait. */ - private final java.util.ArrayDeque events = new java.util.ArrayDeque<>(); + private final EventBuffer events = new EventBuffer(MOST_EVENTS); /** Pickers shown and not yet answered in full (guarded by events). */ private int asked; /** The window's insets as insets() gives them, written on the main thread. */ @@ -57,14 +62,15 @@ public class MainActivity extends NativeActivity { * TrackService is refused, so it is tried again the next time the app comes into sight. */ private boolean keepWanted; - /** Events dropped past MOST_EVENTS since the Rust side last took them (guarded by events). */ - private int dropped; /** Why TrackService couldn't run, until takeEvents says so. */ - private static volatile String refusal; + private static final java.util.concurrent.atomic.AtomicReference refusal = + new java.util.concurrent.atomic.AtomicReference<>(); /** The activity, while it is created: TrackService's refusal reaches it at once. */ private static volatile MainActivity current; /** The listener positions arrive on, while they are asked for. */ private android.location.LocationListener positions; + /** Providers this listener actually registered, on the main thread. */ + private final LocationProviders requestedProviders = new LocationProviders(); /** The user is being asked for leave to use the location, and hasn't answered. */ private boolean asking; /** The activity is in sight (between onStart and onStop). */ @@ -261,7 +267,7 @@ private void startTrackService() { * positions out of sight now, not at the next frame (an app out of sight draws none). */ static void serviceRefused(String why) { - refusal = why; + refusal.set(why); MainActivity activity = current; if (activity != null) { activity.keepLocatingRefused(); @@ -318,11 +324,9 @@ public void onProviderDisabled(String provider) { if (!locating || manager == null) { return; } - for (String other : new String[] {android.location.LocationManager.GPS_PROVIDER, - android.location.LocationManager.NETWORK_PROVIDER}) { - if (manager.getAllProviders().contains(other) && manager.isProviderEnabled(other)) { - return; - } + if (requestedProviders.anyEnabled(other -> manager.getAllProviders().contains(other) + && manager.isProviderEnabled(other))) { + return; } stopped("This device's location was turned off in its settings."); } @@ -341,6 +345,7 @@ public void onStatusChanged(String provider, int status, android.os.Bundle extra if (manager.getAllProviders().contains(provider) && manager.isProviderEnabled(provider) && (!satellites || allowed(android.Manifest.permission.ACCESS_FINE_LOCATION))) { manager.requestLocationUpdates(provider, 1000, 0, positions, getMainLooper()); + requestedProviders.add(provider); asked++; } } catch (SecurityException | IllegalArgumentException e) { @@ -358,6 +363,7 @@ private void stopLocating() { manager.removeUpdates(positions); } positions = null; + requestedProviders.clear(); } @Override @@ -461,20 +467,13 @@ public void save(final String name, final byte[] bytes) { /** What happened since the last call, oldest first, one event to a line. */ public String takeEvents() { - String why = refusal; + String why = refusal.getAndSet(null); if (why != null) { - refusal = null; keepLocatingRefused(); add("failed\t" + why); } synchronized (events) { - if (dropped > 0) { - events.addFirst("failed\t" + dropped + " positions found while RustGIS was out of sight were dropped: more were waiting than it keeps."); - dropped = 0; - } - String taken = String.join("\n", events); - events.clear(); - return taken; + return events.take(); } } @@ -549,12 +548,7 @@ private boolean start(Intent intent, int request) { private void add(String event) { synchronized (events) { - // Out of sight for days, the oldest go first, and how many is said. - if (events.size() >= MOST_EVENTS) { - events.pollFirst(); - dropped++; - } - events.add(event.replaceAll("[\\n\\r]", " ")); + events.add(event); } } @@ -635,7 +629,9 @@ protected void onActivityResult(int request, int result, Intent data) { try { if (dir.mkdirs()) { try { - mark.createNewFile(); + try (FileOutputStream active = new FileOutputStream(mark)) { + active.write(COPY_OWNER.getBytes(java.nio.charset.StandardCharsets.UTF_8)); + } } catch (java.io.IOException e) { // Unmarked, the pick is still kept while a copy's .part is in it. } diff --git a/apps/rustgis-mobile/src/ios.rs b/apps/rustgis-mobile/src/ios.rs index 5e7cd0d44d..40bbd9fb3f 100644 --- a/apps/rustgis-mobile/src/ios.rs +++ b/apps/rustgis-mobile/src/ios.rs @@ -204,8 +204,7 @@ impl PickerDelegate { // The pick's folder is marked while documents are copied into it, for the // whole pick: Remove Unused Copies keeps it, also between two documents. // Unmarked, the pick is still kept while a copy's `.part` is in it. - let mark = picked.join(rustgis_ui_touch::COPYING); - let _ = std::fs::create_dir_all(&picked).and_then(|()| std::fs::write(&mark, b"")); + let copying_mark = rustgis_ui_touch::mark_copying(&picked).ok(); let mut copies = 0usize; let mut sources: Vec> = Vec::new(); for url in urls { @@ -216,7 +215,7 @@ impl PickerDelegate { }); copies += usize::from(copy_in(&copied, &purpose, &url, &picked.join(k.to_string()))); } - let _ = std::fs::remove_file(&mark); + drop(copying_mark); // Every copy failed, each removing its own file and folder: this one is empty // too. if copies == 0 { diff --git a/apps/rustgis-mobile/src/lib.rs b/apps/rustgis-mobile/src/lib.rs index 233b324879..4db7b8633a 100644 --- a/apps/rustgis-mobile/src/lib.rs +++ b/apps/rustgis-mobile/src/lib.rs @@ -9,7 +9,8 @@ //! Not here yet: preferences and the keychain; autosave. #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] -#[cfg(any(target_os = "android", target_os = "ios"))] +#[cfg(any(target_os = "android", target_os = "ios", test))] +#[cfg_attr(test, expect(dead_code, reason = "host tests exercise picker contracts without starting a phone window"))] mod shell; #[cfg(target_os = "android")] diff --git a/apps/rustgis-mobile/src/phone.rs b/apps/rustgis-mobile/src/phone.rs index df4c553c0e..beb2d96697 100644 --- a/apps/rustgis-mobile/src/phone.rs +++ b/apps/rustgis-mobile/src/phone.rs @@ -40,7 +40,13 @@ impl Phone { }) }) .map_err(|e| i18n::fmt("Android's Java side can't be reached: {error}", &[("error", &e.to_string())]))?; - Ok(Phone { activity, unreachable: RefCell::new(None) }) + let phone = Phone { activity, unreachable: RefCell::new(None) }; + let owner = phone.java(|env, activity| { + let token = env.call_method(activity, jni_str!("copyOwner"), jni_sig!(() -> JString), &[])?.l()?; + env.cast_local::(token)?.try_to_string(env) + })?; + rustgis_ui_touch::set_copy_owner(owner)?; + Ok(phone) } /// Runs `call` with the Java environment and the activity, on this thread. diff --git a/apps/rustgis-mobile/src/shell.rs b/apps/rustgis-mobile/src/shell.rs index 9125854eac..3713c5fb8b 100644 --- a/apps/rustgis-mobile/src/shell.rs +++ b/apps/rustgis-mobile/src/shell.rs @@ -56,16 +56,17 @@ type Asked = Rc>; #[derive(Clone)] struct Kept { csv: Option, + shapefile: Option, open: Option<&'static str>, picture: Option>, layout_picture: bool, } impl Kept { - fn of(app: &RustGisApp) -> Kept { - let ui = &app.ui; + fn of(ui: &rustgis_ui_egui::UiState) -> Kept { Kept { csv: ui.pending_csv.clone(), + shapefile: ui.pending_shapefile.clone(), open: ui.pending_open, picture: ui.pending_picture.clone(), layout_picture: ui.pending_layout_picture, @@ -75,6 +76,7 @@ impl Kept { fn restore(self, app: &mut RustGisApp) { let ui = &mut app.ui; ui.pending_csv = self.csv; + ui.pending_shapefile = self.shapefile; ui.pending_open = self.open; ui.pending_picture = self.picture; ui.pending_layout_picture = self.layout_picture; @@ -110,7 +112,7 @@ struct App { /// The requests for files waited on. asked: Asked, /// What the front end kept for that picker, as the frame began ([`Kept`]). - kept: Option, + kept: Rc>>, /// Whether a picker was refused in this frame because one was up. refused_picker: Rc>, /// Files picked for a dialog (purpose, name, copy), until their picker has answered in @@ -164,7 +166,12 @@ impl App

{ self.app.position = None; } } - for event in phone.take_events() { + let events = phone.take_events(); + // Mirrors EventBuffer.PER_FRAME in the Android bridge: a full batch may leave more queued. + if events.len() >= 512 { + ctx.request_repaint(); + } + for event in events { match event { Event::Picked { purpose, name, path } => { // A file answers its own request; one of an earlier request (still being @@ -286,19 +293,20 @@ impl App

{ return; } let (inbox, problems, reading, ctx) = (self.inbox.clone(), self.problems.clone(), self.reading.clone(), ctx.clone()); - reading.fetch_add(1, Ordering::Relaxed); let unread: Vec<(String, PathBuf)> = reads.iter().flat_map(|(_, files)| files.clone()).collect(); - let counted = reading.clone(); - let started = std::thread::Builder::new().name("picked files".into()).spawn(move || { - let mut held = 0u64; - for (purpose, files) in reads { - read_batch(&purpose, files, &inbox, &problems, &mut held); - } - reading.fetch_sub(1, Ordering::Relaxed); - ctx.request_repaint(); + let retire: Vec<_> = + reads.iter().filter(|(purpose, _)| purpose.starts_with("dxf:")).map(|(purpose, _)| crate::mark(purpose)).collect(); + let started = start_reader(&self.inbox, &self.reading, retire, || { + std::thread::Builder::new().name("picked files".into()).spawn(move || { + let mut held = 0u64; + for (purpose, files) in reads { + read_batch(&purpose, files, &inbox, &problems, &mut held); + } + reading.fetch_sub(1, Ordering::Relaxed); + ctx.request_repaint(); + }) }); if let Err(e) = started { - counted.fetch_sub(1, Ordering::Relaxed); let error = e.to_string(); // Nothing will read these copies now: they go, as they would have once read. for (name, path) in unread { @@ -309,6 +317,29 @@ impl App

{ } } +/// Balance admission and retire waiting dialogs even when the OS refuses a worker. +fn start_reader( + inbox: &Inbox, + reading: &AtomicUsize, + retire: Vec, + spawn: impl FnOnce() -> std::io::Result, +) -> std::io::Result { + reading.fetch_add(1, Ordering::Relaxed); + match spawn() { + Ok(worker) => Ok(worker), + Err(error) => { + reading.fetch_sub(1, Ordering::Relaxed); + retire_unread(inbox, retire); + Err(error) + } + } +} + +/// A worker that cannot start answers its DXF requests just as a cancelled picker does. +fn retire_unread(inbox: &Inbox, marks: Vec) { + inbox.lock().unwrap_or_else(|e| e.into_inner()).extend(marks.into_iter().map(|mark| (mark, Vec::new()))); +} + /// Reads one folder's files of a pick (name, copy) into `inbox`, marked for the dialog that /// asked, and removes the copies: their bytes are all that was wanted of them. `held` counts the /// bytes the pick has read so far, across its folders. @@ -367,8 +398,15 @@ fn use_picker( refusals: &Refusals, asked: &Asked, refused_picker: &Rc>, + kept: &Rc>>, ) { let (picker, refused, asking, answers) = (phone.clone(), refusals.clone(), asked.clone(), inbox.clone()); + let candidate = Rc::new(RefCell::new(None)); + let snapshot = candidate.clone(); + app.services.open_options = Some(Box::new(move |ui| { + *snapshot.borrow_mut() = Some(Kept::of(ui)); + })); + let admitted = kept.clone(); let refused_while_up = refused_picker.clone(); app.services.open_async = Some(Box::new(move |purpose: &str| { log::info!("asking for files: {purpose}"); @@ -384,6 +422,10 @@ fn use_picker( true } }; + let options = candidate.borrow_mut().take(); + if !refused_here { + *admitted.borrow_mut() = options; + } let ended = asking.borrow_mut().asked(purpose, refused_here); // Refused because another picker was up: the shell puts back what the front end kept // for that one ([`Kept`]). @@ -431,7 +473,7 @@ impl eframe::App for App

{ // A second picker asked for while one was up was refused: what the front end kept for // the first is put back, so its file is read as it was asked for. if self.refused_picker.replace(false) - && let Some(kept) = self.kept.clone() + && let Some(kept) = self.kept.borrow().clone() { kept.restore(&mut self.app); } @@ -486,7 +528,9 @@ impl eframe::App for App

{ }); } } - self.kept = self.phone.as_ref().is_some_and(|phone| phone.picking()).then(|| Kept::of(&self.app)); + if !self.phone.as_ref().is_some_and(|phone| phone.picking()) { + self.kept.borrow_mut().take(); + } // The pushed files open as files dropped on the desktop window do: once the window // runs, read off its thread with progress and Cancel, each failure said in the window. // Opened before the event loop, a large one would hold the app's start. @@ -603,10 +647,11 @@ pub fn run( ) -> eframe::Result { let (inbox, refusals, asked) = (Inbox::default(), Refusals::default(), Asked::default()); let refused_picker = Rc::new(std::cell::Cell::new(false)); + let kept = Rc::new(RefCell::new(None)); let phone = match phone { Ok(phone) => { let phone = Rc::new(phone); - use_picker(&mut app, &phone, &inbox, &refusals, &asked, &refused_picker); + use_picker(&mut app, &phone, &inbox, &refusals, &asked, &refused_picker, &kept); Some(phone) } Err(e) => { @@ -634,7 +679,7 @@ pub fn run( inbox, refusals, asked, - kept: None, + kept, refused_picker, waiting: Vec::new(), problems: Problems::default(), @@ -643,3 +688,73 @@ pub fn run( }), ) } + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::{Value, json}; + use std::cell::Cell; + + #[derive(Default)] + struct FakePhone(Cell); + impl Platform for FakePhone { + fn pick_files(&self, _: &str) -> Result<(), String> { + self.0.set(true); + Ok(()) + } + fn save_file(&self, _: &str, _: Vec) -> Result<(), String> { + Ok(()) + } + fn locate(&self, _: bool) -> Result<(), String> { + Ok(()) + } + fn keep_locating(&self, _: bool) -> Result<(), String> { + Ok(()) + } + fn take_photo(&self) -> Result<(), String> { + Ok(()) + } + fn take_events(&self) -> Vec { + Vec::new() + } + } + impl Phone for FakePhone { + fn insets(&self, _: f32) -> Option<(egui::SafeAreaInsets, Option)> { + None + } + fn bars_light(&self, _: bool, _: &eframe::Frame) {} + fn busy(&self) -> bool { + self.0.get() + } + } + + #[test] + fn same_frame_refused_picker_restores_the_options_of_the_admitted_request() { + let mut app = RustGisApp::with_sample(); + let phone = Rc::new(FakePhone::default()); + let (inbox, refusals, asked) = (Inbox::default(), Refusals::default(), Asked::default()); + let refused = Rc::new(Cell::new(false)); + let kept = Rc::new(RefCell::new(None)); + use_picker(&mut app, &phone, &inbox, &refusals, &asked, &refused, &kept); + let options = json!({"x": "longitude", "y": "latitude"}); + app.run("ui.addDelimited", options.clone()).unwrap(); + app.run("ui.addData", Value::Null).unwrap(); + assert!(refused.replace(false)); + kept.borrow().clone().unwrap().restore(&mut app); + assert_eq!(app.ui.pending_csv, Some(options)); + // Refusing DXF while the accepted picker is up retires exactly its own request. + app.run("ui.addDxf", Value::Null).unwrap(); + assert_eq!(inbox.lock().unwrap().len(), 1); + assert!(inbox.lock().unwrap()[0].0.starts_with(rustgis_ui_egui::DXF_PICK)); + } + #[test] + fn a_worker_refused_before_reading_retires_its_dxf_answer() { + let inbox = Inbox::default(); + let reading = AtomicUsize::new(0); + let result: std::io::Result<()> = + start_reader(&inbox, &reading, vec![crate::mark("dxf:7")], || Err(std::io::Error::other("injected worker refusal"))); + assert!(result.is_err()); + assert_eq!(reading.load(Ordering::Relaxed), 0); + assert_eq!(*inbox.lock().unwrap(), vec![(crate::mark("dxf:7"), Vec::::new())]); + } +} diff --git a/apps/rustgis-mobile/tests/README.md b/apps/rustgis-mobile/tests/README.md new file mode 100644 index 0000000000..420c6138a8 --- /dev/null +++ b/apps/rustgis-mobile/tests/README.md @@ -0,0 +1,18 @@ +# Phone contract checks + +The host-JVM tests exercise position retention, bounded frame batches, completion order and +requested-provider state without an Android device: + +```sh +mkdir -p target/phone-contract-tests +javac -d target/phone-contract-tests \ + apps/rustgis-mobile/java/org/rustgis/mobile/EventBuffer.java \ + apps/rustgis-mobile/java/org/rustgis/mobile/LocationProviders.java \ + apps/rustgis-mobile/tests/java/org/rustgis/mobile/PhoneContractsTest.java +java -ea -cp target/phone-contract-tests org.rustgis.mobile.PhoneContractsTest +``` + +The shared Rust shell's fake-phone tests cover two picker requests in one frame and injected +worker admission failure. They run on the host through `cargo test -p rustgis-mobile shell::tests`. +The Android activity is also compiled by `build-apk.sh`; `cargo check -p rustgis-mobile --target +aarch64-linux-android` checks the Rust shell against its Android APIs with a configured NDK. diff --git a/apps/rustgis-mobile/tests/java/org/rustgis/mobile/PhoneContractsTest.java b/apps/rustgis-mobile/tests/java/org/rustgis/mobile/PhoneContractsTest.java new file mode 100644 index 0000000000..bbf15215a8 --- /dev/null +++ b/apps/rustgis-mobile/tests/java/org/rustgis/mobile/PhoneContractsTest.java @@ -0,0 +1,48 @@ +package org.rustgis.mobile; + +/** Host-JVM regression checks; run with assertions enabled (no Android runtime needed). */ +public final class PhoneContractsTest { + public static void main(String[] args) { + EventBuffer events = new EventBuffer(1200); + events.add("picked\tdata\told\t/file"); + for (int i = 0; i < 2000; i++) { + events.add("position\t" + i); + } + events.add("ended\tdxf:7"); + int fixes = 0; + int notices = 0; + int completions = 0; + int previous = 799; + while (!events.isEmpty()) { + String[] batch = events.take().split("\n"); + assert batch.length <= EventBuffer.PER_FRAME; + for (String event : batch) { + if (event.startsWith("position\t")) { + int value = Integer.parseInt(event.substring(9)); + assert value == ++previous : event; + fixes++; + } else if (event.startsWith("failed\t")) { + assert event.startsWith("failed\t800 positions"); + notices++; + } else { + assert (completions == 0 && event.startsWith("picked\t")) + || (completions == 1 && event.equals("ended\tdxf:7")); + completions++; + } + } + } + assert fixes == 1200 && notices == 1 && completions == 2; + assert events.take().isEmpty(); + LocationProviders providers = new LocationProviders(); + providers.add("network"); + // Approximate permission registered only network: enabled but unrequested GPS cannot + // keep a dead network listener looking alive. + assert !providers.anyEnabled(provider -> provider.equals("gps")); + assert providers.anyEnabled(provider -> provider.equals("network")); + providers.add("gps"); + assert providers.anyEnabled(provider -> provider.equals("gps")); + providers.clear(); + assert !providers.anyEnabled(provider -> true); + System.out.println("Phone contract checks passed"); + } +} diff --git a/apps/rustgis-python/python/rustgis/__init__.py b/apps/rustgis-python/python/rustgis/__init__.py index a3b467fb18..115e16e49f 100644 --- a/apps/rustgis-python/python/rustgis/__init__.py +++ b/apps/rustgis-python/python/rustgis/__init__.py @@ -370,7 +370,8 @@ def _outputs(self, result: Any) -> Any: A file of several layers (the tables of a GeoPackage, a GDAL file split by geometry type) reports the first as ``"layer"`` and every one as ``"layers"``, so ``"layers"`` - decides. Standalone tables added with the layers are in :meth:`info`, not returned. + decides. Any standalone tables that were actually added are in :meth:`info`, not + returned. GeoPackage attribute-only tables are not imported when adding a whole file. Args: result: The command's result. @@ -562,7 +563,9 @@ def add_data(self, path: PathLike, name: Optional[str] = None, **params: Any) -> path: The file: GeoJSON, CSV, Shapefile (or a zip of them), GeoPackage, GeoParquet, FlatGeobuf, KML/KMZ, GPX, GeoTIFF, ASCII grid, and the formats GDAL or DuckDB read when they are installed. - name: The layer's name (the file's name when left out). + name: The layer's name (the file's name when left out). Native multi-layer imports + apply it only to the first layer. GDAL or DuckDB files split into several layers keep + their generated names and report the ignored name in a notice. **params: More ``layer.addData`` parameters, e.g. ``layer=`` for one table of a GeoPackage, ``x=`` and ``y=`` for a CSV's coordinate columns. diff --git a/apps/rustgis-web/src/web.rs b/apps/rustgis-web/src/web.rs index 397c0bff7f..a2c810a42e 100644 --- a/apps/rustgis-web/src/web.rs +++ b/apps/rustgis-web/src/web.rs @@ -1309,6 +1309,7 @@ fn services(inbox: Inbox, font_inbox: Inbox, notes: Notes, ctx: egui::Context, e ctx.request_repaint(); }); })), + open_options: None, download: Some(Box::new(|name: &str, bytes: Vec| download(name, &bytes))), inbox: Some(inbox), geolibre_manifests: Some(Box::new(move |slot| crate::geolibre::manifests(slot, gl_list_ctx.clone()))), diff --git a/crates/analysis/src/density.rs b/crates/analysis/src/density.rs index c788d22bd2..e957175e90 100644 --- a/crates/analysis/src/density.rs +++ b/crates/analysis/src/density.rs @@ -306,6 +306,10 @@ fn kernel_density_on( return Err("kernel density requires a projected coordinate system with linear units (e.g. meters or feet), not geographic coordinates in degrees".into()); } + if crs.is_geocentric() { + return Err("kernel density requires a projected coordinate system with linear units, not geocentric coordinates".into()); + } + // Determine spatial extent let pt_extent = points_extent(points).ok_or("points have empty extent")?; let base_extent = resolve_grid_extent(&pt_extent, bandwidth, custom_extent)?; @@ -580,6 +584,17 @@ mod tests { assert!(err.contains("requires a projected coordinate system"), "{err}"); } + #[test] + fn public_kernel_density_rejects_geocentric_coordinates() { + let points = [(Coord::new(10.0, 20.0), 1.0)]; + let geocentric = Crs::from_epsg(4978).unwrap(); + assert!(geocentric.is_geocentric()); + let error = crate::kernel_density(&points, None, 1.0, 10.0, KernelType::Quartic, "earth_xyz", geocentric).unwrap_err(); + assert!(error.contains("geocentric"), "{error}"); + let projected = crate::kernel_density(&points, None, 1.0, 10.0, KernelType::Quartic, "projected", Crs::WebMercator).unwrap(); + assert!(projected.bands[0].data.iter().any(|value| *value > 0.0)); + } + #[test] fn non_finite_points_are_skipped() { let pts = vec![(Coord::new(f64::NAN, 100.0), 1.0), (Coord::new(100.0, f64::INFINITY), 1.0), (Coord::new(100.0, 100.0), 1.0)]; diff --git a/crates/analysis/src/tools.rs b/crates/analysis/src/tools.rs index 0e506c89f5..6405f049e4 100644 --- a/crates/analysis/src/tools.rs +++ b/crates/analysis/src/tools.rs @@ -1906,7 +1906,7 @@ fn clip_on(inp: &dyn Inputs, p: &Json, threads: Option, cache_bytes: Opti } r.messages.extend(left.notes("Clip", &clip_ds.name)); if slivers > 0 { - r.messages.push(format!("Clip: {slivers} output feature(s) have less than one millionth of their input feature's area; they may be slivers along the clip boundary (kept unchanged)")); + r.messages.push(format!("Clip: {slivers} output feature(s) have less than one millionth of their input feature's area in the overlay plane; they may be slivers along the clip boundary (kept unchanged)")); } Ok(r) } @@ -5220,11 +5220,12 @@ fn ground_area_perimeter(g: &Geometry) -> (f64, f64) { let polar = ring.0.first().zip(ring.0.last()).is_some_and(|(a, b)| (a.x - b.x).abs() > 180.0); let orientable = !polar && ring.0.iter().all(|c| c.x.is_finite() && (-90.0..=90.0).contains(&c.y)); let polygon = geo::Polygon::new(ring, vec![]); - // Within a longitude span of 180°, the ring cannot enclose more than a hemisphere. + // Within an unwrapped longitude span of 180°, the ring cannot enclose more than a + // hemisphere. // Use signed magnitude there: a tiny overlay sliver can have opposite planar and // geodesic windings, making its unsigned area nearly the whole Earth (#1966). - let wide = polygon.bounding_rect().is_some_and(|bounds| bounds.width() > 180.0); - let (rp, ra) = if orientable && wide { + let wide = orientable && polygon.bounding_rect().is_some_and(|bounds| bounds.width() > 180.0); + let (rp, ra) = if wide { polygon.orient(Direction::Default).geodesic_perimeter_area_unsigned() } else { // Keep smaller local rings and polar caps; invalid coordinates bypass orientation. @@ -13524,7 +13525,7 @@ mod tests { r.messages, [ "Clip: 2 features kept", - "Clip: 2 output feature(s) have less than one millionth of their input feature's area; they may be slivers along the clip boundary (kept unchanged)", + "Clip: 2 output feature(s) have less than one millionth of their input feature's area in the overlay plane; they may be slivers along the clip boundary (kept unchanged)", ] ); let out = layer(r); diff --git a/crates/analysis/src/warp.rs b/crates/analysis/src/warp.rs index 635683ea14..ea1f7d0d8f 100644 --- a/crates/analysis/src/warp.rs +++ b/crates/analysis/src/warp.rs @@ -453,9 +453,11 @@ impl Method { /// Projects `r` onto the grid `g`, through longitude/latitude when their coordinate systems /// differ: each cell sampled at its centre, or (average, mode) taken from every pixel of `r` /// whose centre falls in it. A cell no pixel centre falls in (a grid finer than the raster) takes -/// the pixel under its centre. Cells off the raster are its no-data (else NaN, or 0 in a -/// whole-number type), or see-through for a colour image with neither alpha nor no-data, which -/// gains an alpha band for it. +/// the pixel under its centre. Outside cells use 0 in every band when alpha is present or +/// added; otherwise they use explicit no-data. Without either, they use NaN for floating-point +/// data, RGB data with more than three bands, or when any band has class names, else 0. +/// A three-band RGB image with neither alpha nor +/// no-data gains an alpha band for see-through cells. pub fn project(r: &Raster, g: &Grid, method: Method, name: String) -> Result { project_on(None, r, g, method, name) } diff --git a/crates/analysis/tests/parallel_serial.rs b/crates/analysis/tests/parallel_serial.rs index f9d6992132..cc65ac8b6a 100644 --- a/crates/analysis/tests/parallel_serial.rs +++ b/crates/analysis/tests/parallel_serial.rs @@ -183,6 +183,7 @@ fn outputs(s: &Scene) -> Vec<(String, Made)> { add("dissolve", tool("dissolve", json!({"input": "polygons", "field": "class", "statistics": stats}))); add("near", tool("near", json!({"input": "points", "near": "polygons"}))); add("convex hull", tool("convexHull", json!({"input": "polygons", "group": "None"}))); + add("convex hull all", tool("convexHull", json!({"input": "polygons", "group": "All"}))); add("feature to point", tool("featureToPoint", json!({"input": "polygons"}))); add("simplify", tool("simplify", json!({"input": "polygons", "tolerance": 5}))); add("select by location", tool("selectByLocation", json!({"input": "points", "select": "polygons", "distance": 1}))); diff --git a/crates/assistant/src/chatgpt.rs b/crates/assistant/src/chatgpt.rs index c8caea3474..7ecfc9d190 100644 --- a/crates/assistant/src/chatgpt.rs +++ b/crates/assistant/src/chatgpt.rs @@ -1004,18 +1004,34 @@ mod native { /// The selected registration, including signed-out accounts. Reads the keychain only to move /// an older version's tokens into it (once); otherwise only the file. pub fn load_any() -> Option { - let path = auth_path()?; - let _guard = store(&path).ok()?; - load_any_at(&path, vault()) + let (credentials, warning) = load_any_report(); + if let Some(warning) = warning { + log::warn!("{warning}"); + } + credentials + } + + /// Reads the selected registration and preserves a refused-store diagnostic for the caller. + /// A missing store is a clean signed-out state. A token-store failure may still return the + /// readable registration without tokens, as [`load_any`] does. + pub fn load_any_report() -> (Option, Option) { + let Some(path) = auth_path() else { return (None, None) }; + let _guard = match store(&path) { + Ok(guard) => guard, + Err(e) => return (None, Some(e)), + }; + load_any_report_at(&path, vault()) } + #[cfg(test)] fn load_any_at(path: &Path, v: &Vault) -> Option { + load_any_report_at(path, v).0 + } + + fn load_any_report_at(path: &Path, v: &Vault) -> (Option, Option) { match selected_at(path, v) { - Ok(c) => c, - Err(e) => { - log::warn!("{e}"); - read_accounts(path).ok()?.selected().map(Credentials::without_tokens) - } + Ok(c) => (c, None), + Err(e) => (read_accounts(path).ok().and_then(|accounts| accounts.selected().map(Credentials::without_tokens)), Some(e)), } } @@ -2018,17 +2034,28 @@ mod native { let e = read_accounts(&path).err().unwrap(); assert!(e.to_string().contains("newer RustGIS (version 3; this one reads 2)"), "{e}"); assert!(read_credentials(&path, &v).is_err()); + let (credentials, warning) = load_any_report_at(&path, &v); + assert!(credentials.is_none()); + let warning = warning.unwrap().to_string(); + assert!(warning.contains("newer RustGIS") && !warning.contains("synthetic"), "{warning}"); assert!(write_credentials(&path, &v, &synthetic(now() + 100_000)).is_err()); assert!(sign_out_at(&path, &v).is_err()); assert!(remove_at(&path, &v, "oaiapp_synthetic", "user-1").is_err()); assert_eq!(std::fs::read_to_string(&path).unwrap(), text); // A version that isn't a plain integer is unreadable, not newer-or-not. for odd in [serde_json::json!("3"), serde_json::json!(3.0), serde_json::json!(true)] { - let mut v: Value = serde_json::from_str(&text).unwrap(); - v["version"] = odd; - std::fs::write(&path, v.to_string()).unwrap(); + let mut record: Value = serde_json::from_str(&text).unwrap(); + record["version"] = odd; + std::fs::write(&path, record.to_string()).unwrap(); assert!(read_accounts(&path).is_err()); + let before = std::fs::read(&path).unwrap(); + let (credentials, warning) = load_any_report_at(&path, &v); + assert!(credentials.is_none() && warning.unwrap().to_string().contains("version isn't a whole number")); + assert_eq!(std::fs::read(&path).unwrap(), before); } + let missing = dir.path().join("missing.json"); + let (credentials, warning) = load_any_report_at(&missing, &v); + assert!(credentials.is_none() && warning.is_none()); } #[test] diff --git a/crates/data/src/login_address.rs b/crates/data/src/login_address.rs index 598a584253..39821d282b 100644 --- a/crates/data/src/login_address.rs +++ b/crates/data/src/login_address.rs @@ -95,6 +95,63 @@ pub fn inner(address: &str) -> &str { parse(address).ok().flatten().map_or(address, |b| b.url) } +/// A URL's length with the whitespace/control escapes [`as_sent`] writes counted at their +/// original UTF-8 width. Other escapes keep their byte length. Plain saved URLs use the same +/// rule because project saving removes runtime login wrappers, retaining their encoded URL. +/// Callers also bound the encoded address: this is a limit on the advertised-equivalent URL, +/// not permission to decode arbitrary escapes or to accept an unbounded runtime wrapper. +pub fn advertised_len(address: &str) -> usize { + let url = match parse(address) { + Ok(Some(bound)) => bound.url, + Ok(None) => address, + Err(_) => return address.len(), + }; + let mut chars = url.char_indices(); + let (mut length, mut in_path) = (0usize, true); + while let Some((at, c)) = chars.next() { + if c == '?' || c == '#' { + in_path = false; + } + let escaped = (c == '%').then(|| escaped_char(url.get(at..).unwrap_or_default())).flatten(); + if let Some((decoded, width)) = escaped + && !matches!(decoded, '\t' | '\n' | '\r') + && (decoded.is_whitespace() || decoded.is_control() || (decoded == '\\' && !in_path)) + { + length += decoded.len_utf8(); + for _ in 1..width { + let _ = chars.next(); + } + } else { + length += c.len_utf8(); + } + } + length +} + +/// One percent-encoded UTF-8 character and the bytes its escapes occupy, without decoding the +/// rest of the URL. Four bytes is the most a character can occupy. +fn escaped_char(text: &str) -> Option<(char, usize)> { + let hex = |b: u8| (b as char).to_digit(16).map(|v| v as u8); + let byte = |at: usize| { + let s = text.as_bytes().get(at..at + 3)?; + (s.first() == Some(&b'%')).then_some(())?; + Some(hex(*s.get(1)?)? * 16 + hex(*s.get(2)?)?) + }; + let first = byte(0)?; + let n = match first { + 0..=0x7f => 1, + 0xc2..=0xdf => 2, + 0xe0..=0xef => 3, + 0xf0..=0xf4 => 4, + _ => return None, + }; + let mut bytes = [0; 4]; + for (i, b) in bytes.iter_mut().take(n).enumerate() { + *b = byte(i * 3)?; + } + Some((std::str::from_utf8(bytes.get(..n)?).ok()?.chars().next()?, n * 3)) +} + /// Canonical origin of a plain URL or a well-formed bound URL. pub fn origin(address: &str) -> Option { raw_origin(inner(address)) @@ -192,6 +249,23 @@ pub fn check(address: &str) -> Result<(), &'static str> { mod tests { use super::*; + #[test] + fn advertised_length_counts_only_binding_normalization_escapes() { + let base = bind("0123456789abcdef", "https://maps.test", "https://maps.test/wms").unwrap(); + for raw in ["https://maps.test/a b?name=a\u{2003}b", "https://maps.test/a\\b?q=a\\b", "https://maps.test/x?q=a\u{000b}b"] { + let bound = inherit(&base, raw); + assert_eq!(advertised_len(&bound), raw.len(), "{raw:?}"); + } + for encoded in ["%41", "%2F", "%09", "%0A", "%0D", "%E2%80", "%FF", "%zz", "%5C"] { + let url = format!("https://maps.test/{encoded}"); + let bound = inherit(&base, &url); + assert_eq!(advertised_len(&bound), url.len(), "{encoded}"); + assert_eq!(advertised_len(&url), url.len(), "plain addresses stay unchanged"); + } + let broken = "rustgis-login:broken"; + assert_eq!(advertised_len(broken), broken.len()); + } + #[test] fn binding_length_and_alphabet_are_checked() { let wrapped = |b: &str| format!("{PREFIX}{b}:14:https://a.test:https://a.test/x"); diff --git a/crates/engine/src/cmd/qgis.rs b/crates/engine/src/cmd/qgis.rs index f1c4f749d2..cc5c7e2888 100644 --- a/crates/engine/src/cmd/qgis.rs +++ b/crates/engine/src/cmd/qgis.rs @@ -145,7 +145,7 @@ fn import_project(s: &mut Session, p: &Value) -> CmdResult { let left = report.notes.iter().filter(|n| n.kind == NoteKind::Skipped).count(); s.notify( format!( - "Imported {} of {} QGIS layers; {} things drawn differently, {left} left out (see the import's notes)", + "Imported {} of {} QGIS layers; {} things drawn differently, {left} left out (see Messages)", added.len(), project.layers.len() + usize::from(project.main_annotations.is_some()), report.notes.len() - left @@ -153,6 +153,7 @@ fn import_project(s: &mut Session, p: &Value) -> CmdResult { false, ); } + notify_report(s, &report); Ok(json!({"map": name, "layers": added, "notes": notes})) } @@ -357,7 +358,7 @@ fn add_layer(s: &mut Session, layer: &qgis::ProjectLayer, has_dir: bool, report: } check_fields(l, name, &layer.style.rules, layer.style.categories_by.as_ref(), &layer.style.bool_classes, report); if layer.style.diagram.is_some() { - check_diagram(l, name, report); + layer.style.check_diagram_fields(l, name, report); } bind_time(l, name, layer.style.time.as_deref(), report); Ok(Some(added_name)) @@ -639,36 +640,6 @@ fn fitted(rect: &rustgis_data::Geometry, (w, h): (u32, u32)) -> rustgis_data::Ge ]]]) } -/// Checks the diagram a style set against the layer's fields: a category on a field the layer -/// hasn't is left out, and a diagram left with none is removed, noted with the fields. On a -/// graphics layer, which takes none, the style's is noted as left out. -fn check_diagram(l: &mut rustgis_map::Layer, name: &str, report: &mut Report) { - // A graphics layer takes none (`Style::apply` leaves it out), and its own is left alone. - if l.graphics { - report.skipped(name, "its diagrams (a graphics layer draws none)"); - return; - } - let Some(d) = l.diagram.as_mut() else { return }; - let missing: Vec = d.categories.iter().filter(|c| l.dataset.field_index(&c.field).is_none()).map(|c| c.field.clone()).collect(); - if missing.is_empty() { - return; - } - d.categories.retain(|c| l.dataset.field_index(&c.field).is_some()); - let none_left = d.categories.is_empty(); - if none_left { - l.diagram = None; - } - let shown: Vec<&str> = missing.iter().take(8).map(String::as_str).collect(); - let more = if missing.len() > shown.len() { ", …" } else { "" }; - let what = if none_left { "its diagrams, whose fields the layer hasn't" } else { "its diagram categories on fields the layer hasn't" }; - report.skipped(name, format!("{what} ({}{more})", shown.join(", "))); - // A pie sized between two values is sized by the sum of the categories left, not of all of - // those QGIS adds up. - if l.diagram.as_ref().is_some_and(|d| matches!(d.size, rustgis_map::DiagramSize::Scaled { .. })) { - report.approximated(name, "its diagrams sized by the sum of the categories left, without those on fields the layer hasn't"); - } -} - /// An imported QGIS expression read on `ds`. QGIS measures `$area`, `$perimeter` and `$length` /// in the project's units, which aren't read yet, so an expression using them doesn't read. fn parse_imported(text: &str, ds: &rustgis_data::Dataset) -> Result { @@ -1365,6 +1336,72 @@ fn note_json(n: &qgis::Note) -> Value { json!({"layer": n.layer, "kind": n.kind, "text": n.text, "count": n.count}) } +/// The returned report serves CLI and Python callers; the app also needs every coalesced +/// approximation and omission in its Messages pane. Report bounds and groups these notes. +fn notify_report(s: &mut Session, report: &Report) { + let bounded = |text: &str, limit| { + let mut chars = text.chars(); + let mut text: String = chars.by_ref().take(limit).collect(); + if chars.next().is_some() { + text.push_str("… [truncated]"); + } + text + }; + for kind in [NoteKind::Approximated, NoteKind::Skipped] { + let notes: Vec<_> = report.notes.iter().filter(|note| note.kind == kind).collect(); + let Some(first) = notes.first() else { continue }; + if notes.len() == 1 { + let template = match kind { + NoteKind::Approximated => "QGIS import, {layer}: {detail} (drawn differently; count {count})", + NoteKind::Skipped => "QGIS import, {layer}: {detail} (left out; count {count})", + }; + s.notify_text( + crate::notice_text::NoticeText::format( + template, + [ + ("layer", bounded(if first.layer.is_empty() { "project" } else { &first.layer }, 80).into()), + ("detail", bounded(&first.text, 320).into()), + ("count", first.count.to_string().into()), + ], + ), + false, + ); + continue; + } + // One notification per kind across layers. The command result retains the complete + // report; the app gets a bounded sample and the total, including notes beyond the sample. + let count = notes.iter().fold(0usize, |count, note| count.saturating_add(note.count)); + let layers = notes.iter().map(|note| ¬e.layer).collect::>().len(); + let mut samples = notes + .iter() + .take(8) + .map(|note| { + format!( + "{}: {} (count {})", + bounded(if note.layer.is_empty() { "project" } else { ¬e.layer }, 80), + bounded(¬e.text, 320), + note.count + ) + }) + .collect::>() + .join("; "); + if notes.len() > 8 { + samples.push_str(&format!("; {} additional notes not listed here", notes.len() - 8)); + } + let template = match kind { + NoteKind::Approximated => "QGIS import: {count} approximations; affected layers: {layers}. {samples}", + NoteKind::Skipped => "QGIS import: {count} omissions; affected layers: {layers}. {samples}", + }; + s.notify_text( + crate::notice_text::NoticeText::format( + template, + [("count", count.to_string().into()), ("layers", layers.to_string().into()), ("samples", samples.into())], + ), + false, + ); + } +} + fn import_style(s: &mut Session, p: &Value) -> CmdResult { let text = qgis_text(p)?; let id = s.layer_id(p)?; @@ -1403,7 +1440,7 @@ fn import_style(s: &mut Session, p: &Value) -> CmdResult { style.apply(l); check_fields(l, &name, &style.rules, style.categories_by.as_ref(), &style.bool_classes, &mut report); if style.diagram.is_some() { - check_diagram(l, &name, &mut report); + style.check_diagram_fields(l, &name, &mut report); } // Bound: the layer shows the slider's window at once (nothing changes while it is off). if bind_time(l, &name, style.time.as_deref(), &mut report) { @@ -1414,5 +1451,6 @@ fn import_style(s: &mut Session, p: &Value) -> CmdResult { s.notify(notice, false); } } + notify_report(s, &report); Ok(json!({"notes": report.notes.iter().map(note_json).collect::>()})) } diff --git a/crates/engine/src/cmd/share.rs b/crates/engine/src/cmd/share.rs index c10a4cff91..8f50111301 100644 --- a/crates/engine/src/cmd/share.rs +++ b/crates/engine/src/cmd/share.rs @@ -676,6 +676,22 @@ pub fn read_date_notes(file: &str, layer: &str, lost: &rustgis_map::Unreadable) empty.chain(lost.rounded.iter().map(|n| format!("{file}: {layer}: {n} when read for the export"))).collect() } +/// The complete diagnostics from a whole read made for an export, including metadata losses +/// discovered by a temporary source opening that the live layer never received. +pub fn read_notes(file: &str, layer: &str, lost: &rustgis_map::Unreadable) -> Vec { + read_date_notes(file, layer, lost).into_iter().chain(lost.messages.iter().map(|n| format!("{file}: {n}"))).collect() +} + +/// A counted source Z/M diagnostic replaces only the equivalent generic export warning. +/// Resident/older layers and custom providers without that diagnostic keep their warning. +pub fn coalesce_read_zm(notes: &mut Vec, file: &str, ds: &rustgis_data::Dataset, lost: &rustgis_map::Unreadable) { + if lost.zm_noted + && let Some(generic) = rustgis_data::zm_export_note([ds], file) + { + notes.retain(|note| note != &generic); + } +} + /// What a GeoPackage export of `ds` says of its dates: date-times written in UTC, and text in /// date fields that isn't a date, and rounded to the millisecond; for a FlatGeobuf, its date fields with no values. Nothing for /// another format, or when there was none. @@ -748,7 +764,7 @@ fn write_layer(l: &rustgis_map::Layer, path: &str, asked: Option, gpkg: Gpk let unfit = if parquet { rustgis_data::source::parquet::values_note(ds, path) } else { None }; let null_numbers = rustgis_data::export::numbers_written_as_null(&ds.features, &ds.fields, &ext); let csv = if ext == "csv" { rustgis_data::io::CsvLosses::of(&ds.features, ds.geometry_type) } else { Default::default() }; - let mut dates = read_date_notes(path, &ds.name, &unreadable); + let mut dates = read_notes(path, &ds.name, &unreadable); dates.extend(gpkg_dates_note(ds, &ext, path)); (ds.features.len().saturating_sub(left_out), left_out, unread, table, unfit, null_numbers, None, csv, dates) } @@ -759,6 +775,7 @@ fn write_layer(l: &rustgis_map::Layer, path: &str, asked: Option, gpkg: Gpk // raster layer's extent is its raster's, which the file doesn't hold. let extent = if chunks.is_some() { l.extent() } else { ds.extent() }; let mut notes = rustgis_data::export_notes([(ds, extent)], path, crs); + coalesce_read_zm(&mut notes, path, ds, &unreadable); if asked.is_none() { notes.extend(rustgis_data::io::default_export_note(ds, path)); } diff --git a/crates/engine/src/cmd/whole.rs b/crates/engine/src/cmd/whole.rs index e8d6cf77ed..526fb5f7ba 100644 --- a/crates/engine/src/cmd/whole.rs +++ b/crates/engine/src/cmd/whole.rs @@ -31,6 +31,7 @@ pub(crate) struct Unread { rows: usize, dates: Option, rounded: Option, + messages: Vec, } impl Whole { @@ -72,7 +73,7 @@ impl Whole { impl Unread { /// What a whole read of `layer` couldn't read, to say. pub(crate) fn of(layer: String, unread: Unreadable) -> Self { - Unread { layer, rows: unread.rows, dates: unread.dates, rounded: unread.rounded } + Unread { layer, rows: unread.rows, dates: unread.dates, rounded: unread.rounded, messages: unread.messages } } /// Says the rows left out, as Load All Features says them, the dates read as null and the @@ -87,13 +88,16 @@ impl Unread { if let Some(rounded) = &self.rounded { s.notify(format!("{}: {rounded}", self.layer), false); } + for message in &self.messages { + s.notify(message.clone(), false); + } } /// What [`Unread::say`] says, as lines: for a tool's messages. pub(crate) fn lines(&self) -> Vec { let rows = (self.rows > 0).then(|| unread_notice(&self.layer, self.rows)); let dates = [&self.dates, &self.rounded].into_iter().flatten().map(|n| format!("{}: {n}", self.layer)); - rows.into_iter().chain(dates).collect() + rows.into_iter().chain(dates).chain(self.messages.iter().cloned()).collect() } /// `error`, with the rows left out when there were any: they may be why it failed. @@ -185,3 +189,100 @@ pub(crate) fn layers_read_whole<'a>(s: &super::InputContext<'a>, command: &str, }; layers.into_iter().filter(|l| l.is_streamed()).collect() } + +#[cfg(test)] +mod tests { + use super::*; + + /// A detached read can be the first opening: its source metadata never reaches the live + /// layer, so the command and export must carry the diagnostics with the rows they read. + #[test] + fn detached_whole_reads_relay_new_metadata_and_leave_clean_reads_quiet() { + for (tag, coordinates, lossy) in [("3d", "[1,2,3]", true), ("xy", "[1,2]", false)] { + let path = std::env::temp_dir().join(format!("rustgis-whole-notices-{}-{tag}.geojson", std::process::id())); + let coordinates: serde_json::Value = serde_json::from_str(coordinates).unwrap(); + let text = serde_json::json!({"type":"FeatureCollection","features":[{ + "type":"Feature","properties":{},"geometry":{"type":"Point","coordinates":coordinates} + }]}) + .to_string(); + std::fs::write(&path, text).unwrap(); + let mut layer = Layer::new(1, Dataset::new("sites", rustgis_data::GeometryType::Point, Vec::new())); + let mut source = rustgis_map::source::SourceLayer::default(); + source.path = Some(path.display().to_string()); + layer.source = Some(source); + let (dataset, unread) = Whole::of(&layer).read().unwrap(); + assert_eq!(dataset.features.len(), 1); + assert_eq!(dataset.zm_dropped, lossy); + assert!(layer.source.as_ref().unwrap().info().is_none(), "the live source remains unopened"); + assert_eq!(unread.lines().iter().filter(|n| n.contains("Z or M")).count(), usize::from(lossy)); + let mut session = Session::new(crate::Project::new("notices")); + unread.say(&mut session); + assert_eq!(session.notices.len(), usize::from(lossy)); + let (_, lost) = layer.load_all_counted().unwrap(); + let export = super::super::share::read_notes("out.kml", &layer.name, &lost); + assert_eq!(export.len(), usize::from(lossy)); + assert!(export.iter().all(|n| n.starts_with("out.kml: "))); + std::fs::remove_file(path).unwrap(); + } + } + + /// Commands and tools keep non-date diagnostics alongside the existing specialized dates. + #[test] + fn whole_read_messages_reach_both_session_and_tool_consumers_once() { + let unread = Unread::of( + "sites".into(), + Unreadable { + dates: Some("1 value isn't a date and was read as empty, in 'day'".into()), + messages: vec!["source: conversion left out 2 source features".into()], + ..Default::default() + }, + ); + let mut session = Session::new(crate::Project::new("notices")); + unread.say(&mut session); + assert_eq!(session.notices.iter().map(|n| &n.text).collect::>(), unread.lines().iter().collect::>()); + assert_eq!(session.notices.len(), 2); + assert_eq!(unread.lines().iter().filter(|n| n.contains("conversion left out 2")).count(), 1); + assert!(Unread::of("clean".into(), Unreadable::default()).lines().is_empty()); + } + + /// A fresh counted Z/M report replaces the export's generic loss notice, while a resident + /// layer that lost its heights earlier still needs that generic export warning. + #[test] + fn whole_exports_report_dimension_loss_once_and_keep_resident_warnings() { + let root = std::env::temp_dir().join(format!("rustgis-whole-export-notices-{}", std::process::id())); + std::fs::create_dir_all(&root).unwrap(); + for (tag, coordinates, old_loss, expected) in [ + ("3d", Some(serde_json::json!([1, 2, 3])), false, 1), + ("xy", Some(serde_json::json!([1, 2])), false, 0), + ("resident", None, true, 1), + ] { + let mut ds = Dataset::new("sites", rustgis_data::GeometryType::Point, Vec::new()); + ds.push(Some(rustgis_data::Geometry::Point(rustgis_geom::Coord::new(1.0, 2.0))), Vec::new()); + ds.zm_dropped = old_loss; + let mut layer = Layer::new(1, ds); + if let Some(coordinates) = coordinates { + let input = root.join(format!("{tag}.geojson")); + let text = serde_json::json!({"type":"FeatureCollection","features":[{ + "type":"Feature","properties":{},"geometry":{"type":"Point","coordinates":coordinates} + }]}) + .to_string(); + std::fs::write(&input, text).unwrap(); + let mut source = rustgis_map::source::SourceLayer::default(); + source.path = Some(input.display().to_string()); + layer.source = Some(source); + } + let mut session = Session::new(crate::Project::new("notices")); + let id = session.map_mut().add_layer(layer); + let output = root.join(format!("{tag}.kml")); + session.run("share.exportLayer", &serde_json::json!({"layer":id,"path":output})).unwrap(); + let notes: Vec<_> = session.notices.iter().filter(|n| n.text.contains("Z or M")).collect(); + assert_eq!(notes.len(), expected, "{tag}: {notes:?}"); + if tag == "3d" { + assert!(notes[0].text.contains("1 feature had Z or M"), "the counted reader report survives"); + } else if old_loss { + assert!(notes[0].text.contains("its geometry has no Z or M"), "the older loss is still reported"); + } + } + std::fs::remove_dir_all(root).unwrap(); + } +} diff --git a/crates/engine/src/cmd/workflows.rs b/crates/engine/src/cmd/workflows.rs index 8677c51a64..8f9db99079 100644 --- a/crates/engine/src/cmd/workflows.rs +++ b/crates/engine/src/cmd/workflows.rs @@ -1935,11 +1935,12 @@ pub(crate) fn convert_work(s: &Session, p: &Value) -> Result 0 { notes.push(format!("{path}: {} rows couldn't be read and were left out", unreadable.rows)); } - notes.extend(super::share::read_date_notes(&path, &ds.name, &unreadable)); + notes.extend(super::share::read_notes(&path, &ds.name, &unreadable)); (notes, rustgis_data::export::numbers_written_as_null(&ds.features, &ds.fields, format)) }; let bytes = std::fs::metadata(&path).map_err(|e| format!("{path}: {e}"))?.len(); diff --git a/crates/engine/src/project.rs b/crates/engine/src/project.rs index 509bc78625..db4620d0e8 100644 --- a/crates/engine/src/project.rs +++ b/crates/engine/src/project.rs @@ -2472,6 +2472,28 @@ mod limit_tests { mod login_binding_tests { use super::*; + #[test] + fn a_near_limit_encoded_legend_survives_save_and_recovery_unbinding() { + use rustgis_data::{login_address, ogc::Legend}; + let document = login_address::bind("0123456789abcdef", "https://maps.test", "https://maps.test/wms").unwrap(); + let prefix = "https://maps.test/legend?name="; + let raw = format!("{prefix}{}x", " ".repeat(rustgis_map::tiles::MAX_LEGEND_URL - prefix.len() - 1)); + let bound = login_address::inherit(&document, &raw); + let mut tile = rustgis_map::TileLayer::new("Legend", rustgis_map::tiles::TileKind::Raster, "https://maps.test/{z}/{x}/{y}.png"); + tile.legends = rustgis_map::tiles::checked_legends([Legend { url: bound.clone(), ..Default::default() }]); + assert_eq!(tile.legends.len(), 1); + let mut project = Project::default(); + project.map_mut().tiles.push(tile); + let (saved, _) = project.saved_text(None).unwrap(); + for text in [saved, project.recovery_text().unwrap()] { + assert!(!text.contains(login_address::PREFIX)); + let opened = Project::from_json(&text).unwrap(); + let legends = &opened.map().tiles[0].legends; + assert_eq!(legends.len(), 1, "unbinding must not make an accepted legend too long"); + assert_eq!(legends[0].url, login_address::inner(&bound)); + } + } + #[test] fn saves_and_imports_strip_runtime_capabilities_including_recorded_urls() { let url = rustgis_data::login_address::bind( diff --git a/crates/engine/src/tests.rs b/crates/engine/src/tests.rs index 79d1154b86..9394fd1d99 100644 --- a/crates/engine/src/tests.rs +++ b/crates/engine/src/tests.rs @@ -1209,6 +1209,7 @@ fn an_export_says_what_its_whole_read_did_to_dates() { rows: 0, dates: Some("1 value isn't a date and was read as empty, in 'day'".into()), rounded: Some("1 date-time was rounded to the microsecond, as more digits than that aren't kept, in 'at'".into()), + ..Default::default() }; assert_eq!( crate::cmd::share::read_date_notes("out.geojson", "Sites", &lost), @@ -2553,6 +2554,11 @@ fn qgis_diagrams_import_on_the_fields_the_layer_has() { let r = s.run("style.importQgis", &json!({"layer": "Countries", "text": scaled(pair)})).unwrap(); assert!(notes(&r).iter().any(|n| n == sized), "{:?}", notes(&r)); assert!(!notes(&r).iter().any(|n| n.contains("not by")), "the sum QGIS sizes by is the categories': {:?}", notes(&r)); + // A missing category excluded by QGIS's classification does not change the sizing sum. + let matching = scaled(pair).replace(""pop_est" + "nosuch"", ""pop_est""); + let r = s.run("style.importQgis", &json!({"layer": "Countries", "text": matching})).unwrap(); + assert!(notes(&r).iter().any(|n| n.contains("fields the layer hasn't (nosuch)")), "{:?}", notes(&r)); + assert!(!notes(&r).iter().any(|n| n.contains("not by") || n == sized), "the retained sum matches QGIS: {:?}", notes(&r)); let r = s.run("style.importQgis", &json!({"layer": "Countries", "text": qml(pair)})).unwrap(); assert!(!notes(&r).iter().any(|n| n == sized), "{:?}", notes(&r)); let r = s @@ -2566,6 +2572,59 @@ fn qgis_diagrams_import_on_the_fields_the_layer_has() { assert!(s.map().layer_by_name("Sketch").unwrap().diagram.is_none()); } +#[test] +fn qgis_import_reports_reach_session_notices_without_clean_import_noise() { + let mut s = world(); + for field in ["pop_est", "missing_category"] { + let mark = s.notice_mark(); + let qml = format!( + r##""## + ); + let result = s.run("style.importQgis", &json!({"layer":"Countries", "text":qml})).unwrap(); + let notes = result["notes"].as_array().unwrap(); + let said = s.notices_since(mark); + assert_eq!(said.len(), notes.len()); + for note in notes { + assert!(said.iter().any(|n| n.text.contains(note["text"].as_str().unwrap()) && n.text.contains("count 1"))); + } + assert_eq!(said.is_empty(), field == "pop_est"); + } + // Project imports use the same delivery path, including notes for a layer left out. + for layers in [ + "", + r#"xUnsupportednot-a-providerx"#, + ] { + let mark = s.notice_mark(); + let result = s.run("project.importQgis", &json!({"text":format!("{layers}")})).unwrap(); + let said = s.notices_since(mark); + for note in result["notes"].as_array().unwrap() { + assert!(said.iter().any(|n| n.text.contains(note["text"].as_str().unwrap()) && n.text.contains("count 1"))); + } + assert_eq!(said.is_empty(), layers.is_empty()); + } +} + +#[test] +fn qgis_import_notices_group_layers_and_bound_the_sample_without_dropping_returned_notes() { + let mut s = world(); + let layers = (0..10).map(|i| format!(r#"{i}Unsupported{i}not-a-providerx"#)).collect::(); + let mark = s.notice_mark(); + let result = s.run("project.importQgis", &json!({"text":format!("{layers}")})).unwrap(); + let notes = result["notes"].as_array().unwrap(); + assert_eq!(notes.len(), 10, "the returned report remains complete"); + assert!(notes.iter().all(|note| note["kind"] == "skipped")); + let said: Vec<_> = s.notices_since(mark).iter().filter(|n| n.text.starts_with("QGIS import")).collect(); + assert_eq!(said.len(), 1, "one omission notice across all layers"); + assert!(said[0].text.contains("10 omissions") && said[0].text.contains("affected layers: 10")); + assert!( + said[0].text.contains(notes[0]["layer"].as_str().unwrap()) && said[0].text.contains("2 additional notes not listed here"), + "{}", + said[0].text + ); + assert!(!said[0].text.contains(notes[9]["layer"].as_str().unwrap())); + assert!(said[0].text.len() < 4096); +} + /// A QGIS categorized renderer by an expression classes by it where the expression reads on the /// layer; where it doesn't, every feature draws with the first category's symbol, noted, as before. #[test] diff --git a/crates/geolibre/src/install.rs b/crates/geolibre/src/install.rs index 2197803a0e..e3c9c6fb66 100644 --- a/crates/geolibre/src/install.rs +++ b/crates/geolibre/src/install.rs @@ -254,6 +254,11 @@ mod tests { assert_eq!(crate::runner::version_of(&dest).as_deref(), Some("geolibre-cli 1.8.1")); assert!(!staged.exists() && !tmp.0.join(format!("{}.old", crate::runner::exe_name())).exists()); } +} + +#[cfg(test)] +mod release_tests { + use super::*; /// The installer asks for the pinned release, never the latest one. #[test] diff --git a/crates/geolibre/src/runner.rs b/crates/geolibre/src/runner.rs index fa667639d6..83c8f9b551 100644 --- a/crates/geolibre/src/runner.rs +++ b/crates/geolibre/src/runner.rs @@ -149,11 +149,11 @@ fn output_within(c: Command, limit: Duration) -> Option { fn spawn_when_free(c: &mut Command, stop: &dyn Fn() -> Option) -> Result { let start = Instant::now(); loop { + if let Some(why) = stop() { + return Err(why); + } match c.spawn() { Err(e) if e.kind() == std::io::ErrorKind::ExecutableFileBusy && start.elapsed() < Duration::from_secs(1) => { - if let Some(why) = stop() { - return Err(why); - } std::thread::sleep(Duration::from_millis(10)); } Err(e) => return Err(e.to_string()), @@ -393,25 +393,6 @@ fn stop_reason(live: &dyn Fn() -> Result<(), String>) -> Option { mod tests { use super::*; - /// Runner versions 1.x are used; anything else, or a version that can't be read, isn't, and - /// the reason names the runner, its version and the supported range. - #[test] - fn supported_runner_versions() { - for ok in ["geolibre-cli 1.0.0", "geolibre-cli 1.8.0", "geolibre-cli v1.8.2", "geolibre-cli 1.9.0-dev", "geolibre-cli 1.99.3+abc"] { - assert_eq!(check_supported("r", ok), Ok(()), "{ok}"); - } - for bad in ["geolibre-cli 2.0.0", "geolibre-cli 2.0.0-rc.1", "geolibre-cli 0.11.0"] { - let e = check_supported("/opt/geolibre", bad).unwrap_err(); - assert!(e.contains("/opt/geolibre") && e.contains(bad) && e.contains("1.0.0") && e.contains("2.0.0"), "{e}"); - } - for unreadable in ["geolibre-cli", "geolibre-cli 1.8", "geolibre-cli 1.8.0.1", "geolibre-cli one"] { - assert!(check_supported("r", unreadable).unwrap_err().contains("can't read"), "{unreadable}"); - } - assert!(download_url().ends_with(&format!("/releases/tag/v{RELEASE}"))); - let pinned = parse_version(&format!("geolibre-cli {RELEASE}")).unwrap(); - assert!(pinned >= SUPPORTED_FROM && pinned < SUPPORTED_BEFORE, "the pinned release is supported"); - } - /// Discovery passes over a runner outside the supported range for the next supported one, /// and says why; with only an unsupported one, there is no runner but the reason is given. #[test] @@ -492,3 +473,36 @@ mod tests { assert_eq!(with_still(std::sync::Arc::new(|| true), || stop_reason(&fine)), None); } } + +#[cfg(test)] +mod portable_tests { + use super::*; + + /// A stopped run is refused before attempting even an invalid executable. This also keeps + /// every busy-file retry behind the same cancellation check. + #[test] + fn stopped_run_never_attempts_to_spawn() { + let mut command = Command::new(""); + let error = spawn_when_free(&mut command, &|| Some("cancelled before spawn".to_string())).unwrap_err(); + assert_eq!(error, "cancelled before spawn"); + } + + /// Runner versions 1.x are used; anything else, or a version that can't be read, isn't, and + /// the reason names the runner, its version and the supported range. + #[test] + fn supported_runner_versions() { + for ok in ["geolibre-cli 1.0.0", "geolibre-cli 1.8.0", "geolibre-cli v1.8.2", "geolibre-cli 1.9.0-dev", "geolibre-cli 1.99.3+abc"] { + assert_eq!(check_supported("r", ok), Ok(()), "{ok}"); + } + for bad in ["geolibre-cli 2.0.0", "geolibre-cli 2.0.0-rc.1", "geolibre-cli 0.11.0"] { + let e = check_supported("/opt/geolibre", bad).unwrap_err(); + assert!(e.contains("/opt/geolibre") && e.contains(bad) && e.contains("1.0.0") && e.contains("2.0.0"), "{e}"); + } + for unreadable in ["geolibre-cli", "geolibre-cli 1.8", "geolibre-cli 1.8.0.1", "geolibre-cli one"] { + assert!(check_supported("r", unreadable).unwrap_err().contains("can't read"), "{unreadable}"); + } + assert!(download_url().ends_with(&format!("/releases/tag/v{RELEASE}"))); + let pinned = parse_version(&format!("geolibre-cli {RELEASE}")).unwrap(); + assert!(pinned >= SUPPORTED_FROM && pinned < SUPPORTED_BEFORE, "the pinned release is supported"); + } +} diff --git a/crates/geom/src/lib.rs b/crates/geom/src/lib.rs index a67247de86..3f10c2dcb7 100644 --- a/crates/geom/src/lib.rs +++ b/crates/geom/src/lib.rs @@ -562,7 +562,7 @@ fn wgs84_parallel_m(lat_deg: f64, dlon_deg: f64) -> f64 { WGS84_A / (1.0 - WGS84_E2 * s * s).sqrt() * c.abs() * dlon_deg.to_radians().abs() } -/// Coordinates closer than this (degrees, about 11 m, half a unit in the fourth decimal) are on one +/// Coordinates closer than this (degrees, about 11 m, one unit in the fourth decimal) are on one /// parallel or meridian, for [`ring_lon_steps`]: a band written with coordinates rounded to four /// decimals or more keeps its full turns. const NEAR_DEG: f64 = 1e-4; @@ -1739,6 +1739,7 @@ mod tests { assert_eq!(format_distance(-99_949.0), "-99.9 km"); assert_eq!(format_distance(-99_960.0), "-100 km"); assert_eq!(format_area(-0.3), "0 m²"); + assert_eq!(format_area(-0.5), "0 m²"); assert_eq!(format_area(-0.0), "0 m²"); assert_eq!(format_area(-0.6), "-1 m²"); assert_eq!(format_area(-999_999.4), "-999999 m²"); diff --git a/crates/geom/src/triangulate.rs b/crates/geom/src/triangulate.rs index dc9abe12f8..b73f242e41 100644 --- a/crates/geom/src/triangulate.rs +++ b/crates/geom/src/triangulate.rs @@ -189,7 +189,7 @@ mod tests { assert!(!triangulate_reporting(&l, &[]).unwrap().1); let inner = vec![vec![(1.0, 1.0), (2.0, 1.0), (2.0, 2.0), (1.0, 2.0)]]; assert!(!triangulate_reporting(&sq, &inner).unwrap().1); - // A hole that stays clear of the left side is fine. + // A hole touching the left side at points, without sharing an edge, is fine. assert!(!triangulate_reporting(&sq, &[vec![(0.0, 1.0), (2.0, 2.0), (0.0, 3.0)]]).unwrap().1); } diff --git a/crates/map/src/layout.rs b/crates/map/src/layout.rs index 791c4a8033..89da4399e3 100644 --- a/crates/map/src/layout.rs +++ b/crates/map/src/layout.rs @@ -182,12 +182,12 @@ pub fn utm_epsg(lon: f64, lat: f64) -> u32 { } /// The WGS 84 UTM zone a coordinate system names by its EPSG code (`EPSG:32632`, `32632`, any -/// case): its zone and whether it is the northern hemisphere's. `None` for any other system, and -/// for a UTM zone given as WKT or a PROJ string, which MGRS labels don't read. +/// case, also `epsg 32632`): its zone and whether it is the northern hemisphere's. `None` for any +/// other system, and for a UTM zone given as WKT or a PROJ string, which MGRS labels don't read. pub fn utm_zone_of(crs: &str) -> Option<(u8, bool)> { let t = crs.trim(); let digits = match t.get(..5) { - Some(p) if p.eq_ignore_ascii_case("EPSG:") => t.get(5..)?.trim(), + Some(p) if p.eq_ignore_ascii_case("EPSG:") || p.eq_ignore_ascii_case("EPSG ") => t.get(5..)?.trim(), _ => t, }; if !(digits.len() == 5 && digits.bytes().all(|b| b.is_ascii_digit())) { @@ -2050,6 +2050,8 @@ mod tests { (" epsg:32760 ", Some((60, false))), ("32601", Some((1, true))), ("EPSG: 32733", Some((33, false))), + ("epsg 32632", Some((32, true))), + (" EPSG 32733 ", Some((33, false))), ("EPSG:32600", None), ("EPSG:32661", None), ("EPSG:32761", None), diff --git a/crates/map/src/lib.rs b/crates/map/src/lib.rs index 04007a965b..7d86bb5219 100644 --- a/crates/map/src/lib.rs +++ b/crates/map/src/lib.rs @@ -154,6 +154,33 @@ pub struct Unreadable { /// What it says of date-times rounded to the microsecond /// ([`rustgis_data::source::rounded_dates_note`]). pub rounded: Option, + /// Other reader diagnostics, already qualified by their source. Date diagnostics stay in + /// their typed fields above so consumers can describe what an export did to them once. + pub messages: Vec, + /// `messages` contains the source's Z/M loss diagnostic. An export can keep that more + /// specific report instead of also emitting its generic already-dropped-Z/M warning. + pub zm_noted: bool, +} + +/// Recognizes only diagnostics produced by the common Z/M formatter and the two positive +/// GDAL conversion templates. Unknown/custom wording cannot suppress an export warning. +fn reader_noted_zm(note: &str) -> bool { + if note == rustgis_data::zm_note(None) { + return true; + } + if let Some(count) = note.split_once(' ').and_then(|(n, _)| n.parse::().ok()) + && count > 0 + && note == rustgis_data::zm_note(Some(count)) + { + return true; + } + let Some((_, detail)) = note.strip_prefix("GDAL: '").and_then(|n| n.rsplit_once("' ")) else { return false }; + ["declares", "contains"].into_iter().any(|verb| { + detail + .strip_prefix(&format!("{verb} Z or M coordinates on ")) + .and_then(|n| n.strip_suffix(" source feature(s); conversion keeps only XY coordinates")) + .is_some_and(|n| n == "an unknown number of" || n.parse::().is_ok_and(|count| count > 0 && n == count.to_string())) + }) } /// A layer: a dataset drawn with a renderer and labels. @@ -785,15 +812,30 @@ impl Layer { /// [`Layer::load_all`], with what of the source couldn't be read. pub fn load_all_counted(&self) -> Result<(Dataset, Unreadable), String> { - let (ds, report) = self.load_all_reported()?; - Ok((ds, Unreadable { rows: report.skipped, dates: report.unread_dates, rounded: report.rounded_dates })) + let (ds, report, zm_noted) = self.load_all_with_notices()?; + // materialize also puts the typed date diagnostics in messages. Existing consumers + // give them their own export wording, so preserve only the other messages here. + let date_notes: Vec<_> = report.unread_dates.iter().chain(&report.rounded_dates).map(|n| format!(": {n}")).collect(); + let mut seen = std::collections::HashSet::new(); + let messages = report + .messages + .into_iter() + .filter(|m| !date_notes.iter().any(|date| m.ends_with(date))) + .filter(|m| seen.insert(m.clone())) + .collect(); + Ok((ds, Unreadable { rows: report.skipped, dates: report.unread_dates, rounded: report.rounded_dates, messages, zm_noted })) } /// A whole read with its complete reader report, including messages beyond rows and dates. /// Callers decide which opening notices were already shown and which the new read discovered. pub fn load_all_reported(&self) -> Result<(Dataset, rustgis_data::ReadReport), String> { - let Some(copy) = self.open_source()? else { return Ok((self.dataset.clone(), rustgis_data::ReadReport::default())) }; + self.load_all_with_notices().map(|(ds, report, _)| (ds, report)) + } + + fn load_all_with_notices(&self) -> Result<(Dataset, rustgis_data::ReadReport, bool), String> { + let Some(copy) = self.open_source()? else { return Ok((self.dataset.clone(), rustgis_data::ReadReport::default(), false)) }; let (mut ds, mut report) = copy.read_all()?; + let mut zm_noted = false; if let Some(info) = copy.info() { ds.zm_dropped |= info.zm_dropped; for note in &info.notes { @@ -801,6 +843,7 @@ impl Layer { if !report.messages.contains(&message) { report.messages.push(message); } + zm_noted |= info.zm_dropped && reader_noted_zm(note); } } ds.name = self.dataset.name.clone(); @@ -820,7 +863,7 @@ impl Layer { } } ds.source = self.dataset.source.clone(); - Ok((ds, report)) + Ok((ds, report, zm_noted)) } /// A streamed layer's source, to export whole a chunk at a time, with a dataset of no @@ -3190,6 +3233,27 @@ mod tests { use rustgis_data::{Field, FieldType, Value}; use rustgis_geom::lonlat_to_merc; + #[test] + fn only_canonical_reader_dimension_notices_can_replace_export_warnings() { + for count in [Some(1), Some(25), None] { + assert!(reader_noted_zm(&rustgis_data::zm_note(count))); + } + for detail in + ["declares Z or M coordinates on 2", "contains Z or M coordinates on 1", "declares Z or M coordinates on an unknown number of"] + { + assert!(reader_noted_zm(&format!("GDAL: 'a layer' {detail} source feature(s); conversion keeps only XY coordinates"))); + } + for unknown in [ + "a custom provider has a Z/M flag", + "the source couldn't be checked for Z or M values", + "GDAL: 'x' contains Z or M coordinates on perhaps source feature(s); conversion keeps only XY coordinates", + "GDAL: 'x' contains Z or M coordinates on 0 source feature(s); conversion keeps only XY coordinates", + ] { + assert!(!reader_noted_zm(unknown)); + } + assert!(!reader_noted_zm(&rustgis_data::zm_note(Some(0)))); + } + /// A text field an old project lists for a source's date column becomes a date field, unless /// a filter reads it as text and wouldn't as dates: then it stays text and the filter works. #[test] diff --git a/crates/map/src/qgis/diagram.rs b/crates/map/src/qgis/diagram.rs index 6d3cd3fa6b..87439a63d0 100644 --- a/crates/map/src/qgis/diagram.rs +++ b/crates/map/src/qgis/diagram.rs @@ -17,13 +17,19 @@ use roxmltree::Node; use super::{Report, child, color, points, renderer::field_name}; use crate::Rgba; use crate::diagram::{ - DiagramCategory, DiagramKind, DiagramLabel, DiagramSettings, DiagramSize, MAX_CATEGORIES, MAX_SIZE, MAX_STROKE_WIDTH, + DiagramCategory, DiagramKind, DiagramLabel, DiagramSettings, DiagramSize, MAX_CATEGORIES, MAX_SIZE, MAX_STROKE_WIDTH, MIN_SIZE, }; /// The layer's diagram from its `<…DiagramRenderer>` (a child of a project's `` or a /// style's ``), or `None` when it has none, it is switched off or nothing of it can be /// drawn (noted). `reference` is the layer's reference scale (RustGIS's), for sizes in map units. -pub(super) fn read(node: Node, layer: &str, reference: Option, report: &mut Report) -> Option { +pub(super) fn read( + node: Node, + layer: &str, + reference: Option, + classification: &mut Option, + report: &mut Report, +) -> Option { let r = node.children().find(|c| c.is_element() && c.tag_name().name().ends_with("DiagramRenderer"))?; let cats: Vec = r.children().filter(|c| c.has_tag_name("DiagramCategory")).collect(); // Switched off: QGIS draws nothing, and neither does RustGIS. @@ -124,10 +130,9 @@ pub(super) fn read(node: Node, layer: &str, reference: Option, report: &mut .filter_map(|k| r.attribute(k).map(str::trim)) .find(|v| !v.is_empty()) .unwrap_or_default(); + *classification = Some(by.to_string()); if !sums_categories(by, &categories) { - let what = - if by.is_empty() { "a field by its number".to_string() } else { format!("`{}`", by.chars().take(80).collect::()) }; - report.approximated(layer, format!("its diagrams sized by the sum of their categories, not by {what}")); + report.approximated(layer, sizing_note(by)); } // QGIS grows the pie's area from the lower value's, RustGIS from nothing: the two agree // when the range starts at 0. @@ -176,10 +181,16 @@ pub(super) fn read(node: Node, layer: &str, reference: Option, report: &mut } } - let largest = match size { - DiagramSize::Fixed(s) => s, - DiagramSize::Scaled { min_size, max_size, .. } => min_size.max(max_size), + let (smallest, largest) = match size { + DiagramSize::Fixed(s) => (s, s), + DiagramSize::Scaled { min_size, max_size, .. } => (min_size.min(max_size), min_size.max(max_size)), }; + if smallest < MIN_SIZE { + report.approximated( + layer, + format!("its diagrams smaller than RustGIS draws them, drawn at least {:.2} mm across", MIN_SIZE * 2.0 * 25.4 / 96.0), + ); + } if largest > MAX_SIZE { report.approximated( layer, @@ -206,6 +217,42 @@ pub(super) fn read(node: Node, layer: &str, reference: Option, report: &mut (!d.categories.is_empty()).then_some(d) } +fn sizing_note(by: &str) -> String { + let what = if by.is_empty() { "a field's value".to_string() } else { format!("`{}`", by.chars().take(80).collect::()) }; + format!("its diagrams sized by the sum of their categories, not by {what}") +} + +pub(super) fn check_fields(layer: &mut crate::Layer, name: &str, by: Option<&str>, report: &mut Report) { + if layer.graphics { + report.skipped(name, "its diagrams (a graphics layer draws none)"); + return; + } + let Some(d) = layer.diagram.as_mut() else { return }; + let missing: Vec = + d.categories.iter().filter(|c| layer.dataset.field_index(&c.field).is_none()).map(|c| c.field.clone()).collect(); + if missing.is_empty() { + return; + } + d.categories.retain(|c| layer.dataset.field_index(&c.field).is_some()); + let none_left = d.categories.is_empty(); + let matches_size = by.is_some_and(|by| sums_categories(by, &d.categories)); + let scaled = matches!(d.size, DiagramSize::Scaled { .. }); + if none_left { + layer.diagram = None; + } + let shown: Vec<&str> = missing.iter().take(8).map(String::as_str).collect(); + let more = if missing.len() > shown.len() { ", …" } else { "" }; + let what = if none_left { "its diagrams, whose fields the layer hasn't" } else { "its diagram categories on fields the layer hasn't" }; + report.skipped(name, format!("{what} ({}{more})", shown.join(", "))); + if scaled && (none_left || matches_size) { + if let Some(by) = by { + report.forget(name, super::NoteKind::Approximated, &sizing_note(by)); + } + } else if scaled { + report.approximated(name, "its diagrams sized by the sum of the categories left, without those on fields the layer hasn't"); + } +} + /// Whether `by` (a field, or an expression adding fields) is the sum of exactly the categories' /// fields, which is what a RustGIS diagram is sized by. fn sums_categories(by: &str, categories: &[DiagramCategory]) -> bool { diff --git a/crates/map/src/qgis/mod.rs b/crates/map/src/qgis/mod.rs index 89ac4d00ba..1e6ac4457e 100644 --- a/crates/map/src/qgis/mod.rs +++ b/crates/map/src/qgis/mod.rs @@ -98,6 +98,20 @@ impl Report { pub fn skipped(&mut self, layer: &str, text: impl Into) { self.note(layer, NoteKind::Skipped, text); } + + /// Withdraws a provisional note once checking against the destination fields disproves it. + fn forget(&mut self, layer: &str, kind: NoteKind, text: &str) { + let key = (layer.to_string(), kind, text.to_string()); + let Some(&at) = self.index.get(&key) else { return }; + if let Some(note) = self.notes.get_mut(at) + && note.count > 1 + { + note.count -= 1; + return; + } + self.notes.retain(|n| n.layer != layer || n.kind != kind || n.text != text); + self.index = self.notes.iter().enumerate().map(|(i, n)| ((n.layer.clone(), n.kind, n.text.clone()), i)).collect(); + } } /// A layer's look as a QGIS style has it, in RustGIS's terms. Each part is `None` (or empty) when @@ -120,6 +134,9 @@ pub struct Style { pub labels: Option, /// Its diagram: a pie or bar chart on each feature. pub diagram: Option, + /// The scaled diagram's original classification, for checking once the destination fields + /// are known. Import metadata only, not a saved diagram setting. + pub diagram_by: Option, /// 0 opaque to 100 invisible. pub transparency: Option, /// The field whose dates the time slider reads (QGIS's time settings). Not applied by @@ -331,6 +348,12 @@ pub fn rules_expression(rules: &[RuleFilter]) -> String { } impl Style { + /// Checks an imported diagram against the destination's fields. Its size classification + /// is compared with the categories actually retained, so provisional import notes agree. + pub fn check_diagram_fields(&self, layer: &mut crate::Layer, name: &str, report: &mut Report) { + diagram::check_fields(layer, name, self.diagram_by.as_deref(), report); + } + /// Gives `layer` this style's parts (those it sets). pub fn apply(&self, layer: &mut crate::Layer) { if let Some(t) = self.transparency { @@ -676,7 +699,7 @@ fn read_style_node(node: Node, layer: &str, report: &mut Report) -> Style { if let Some(r) = child(node, "renderer-v2") { renderer::read(r, layer, &mut style, report); } - style.diagram = diagram::read(node, layer, style.reference_scale, report); + style.diagram = diagram::read(node, layer, style.reference_scale, &mut style.diagram_by, report); // A raster's renderer: its opacity, and its colouring as RustGIS's raster style. if let Some(r) = child(node, "pipe").and_then(|p| child(p, "rasterrenderer")) { if let Some(o) = r.attribute("opacity").and_then(|o| o.trim().parse::().ok()).filter(|o| o.is_finite() && *o < 1.0) { diff --git a/crates/map/src/qgis/structure.rs b/crates/map/src/qgis/structure.rs index 8253c65b82..b98b12d560 100644 --- a/crates/map/src/qgis/structure.rs +++ b/crates/map/src/qgis/structure.rs @@ -48,6 +48,7 @@ pub fn bookmarks(root: Node, report: &mut Report) -> Vec { // Those whose rotation doesn't read, facing north. let mut unturned = Vec::new(); let mut groups: Vec<&str> = Vec::new(); + let mut seen_groups = std::collections::HashSet::new(); // Those with no name, named "Bookmark". let mut unnamed = 0usize; // Those whose own coordinate system is there but doesn't read (no code, no PROJ string): @@ -66,7 +67,7 @@ pub fn bookmarks(root: Node, report: &mut Report) -> Vec { continue; } if let Some(g) = b.attribute("group").map(str::trim).filter(|g| !g.is_empty()) - && !groups.contains(&g) + && seen_groups.insert(g) { groups.push(g); } diff --git a/crates/map/src/qgis/tests.rs b/crates/map/src/qgis/tests.rs index b0fb6c515d..d030ec4356 100644 --- a/crates/map/src/qgis/tests.rs +++ b/crates/map/src/qgis/tests.rs @@ -2105,6 +2105,26 @@ fn a_subset_string_reading_the_scale_is_left_out() { } } +#[test] +fn a_resolved_diagram_note_keeps_other_same_named_layers_notes() { + let mut report = Report::default(); + let styles = ["value", "angle"].map(|field| { + let xml = format!( + r##""## + ); + read_style(&xml, "same", &mut report).unwrap() + }); + assert_eq!(report.notes.iter().find(|n| n.text.contains("not by")).unwrap().count, 2); + for (style, field) in styles.iter().zip(["value", "angle"]) { + let mut data = rustgis_data::Dataset::new("same", rustgis_data::GeometryType::Point, Vec::new()); + data.fields.push(rustgis_data::Field::new(field, rustgis_data::FieldType::Double)); + let mut layer = crate::Layer::new(1, data); + style.apply(&mut layer); + style.check_diagram_fields(&mut layer, "same", &mut report); + } + assert_eq!(report.notes.iter().find(|n| n.text.contains("not by")).unwrap().count, 1, "only one layer's provisional note is withdrawn"); +} + #[test] fn diagrams_carry_over_as_pie_diagrams() { use crate::{DiagramKind, DiagramSize}; @@ -2145,11 +2165,12 @@ fn diagrams_carry_over_as_pie_diagrams() { assert_eq!(fields(&s.diagram.unwrap()), ["id", "value"]); assert!(said(&report, r#"1 diagram category by an expression (the first: `"angle" / 2`)"#), "{:?}", report.notes); - // Sized between two values by the sum of its categories, as RustGIS sizes it: not noted. + // Sized by its categories, with the zero lower radius raised to the visible minimum: noted. let (s, report) = read("pie-scaled"); let DiagramSize::Scaled { min_size, max_size, min, max } = s.diagram.unwrap().size else { panic!("not scaled") }; assert!(min_size == crate::diagram::MIN_SIZE && (max_size - mm(10.0)).abs() < 1e-3 && min == 0.0 && max == 400.0); - assert!(report.notes.is_empty(), "{:?}", report.notes); + assert!(said(&report, "smaller than RustGIS draws them"), "{:?}", report.notes); + assert!(!said(&report, "not by"), "{:?}", report.notes); // Histograms and stacked bars are left out, noted: RustGIS's bars draw shares, not values. for (name, part) in [ @@ -2225,7 +2246,14 @@ fn diagrams_carry_over_as_pie_diagrams() { )); assert!(matches!(d.unwrap().size, DiagramSize::Fixed(r) if r == crate::diagram::MAX_SIZE)); assert!(said(&report, "larger than RustGIS draws them") && said(&report, "outline, wider"), "{:?}", report.notes); - // Between two values from a lower value other than 0: noted; from 0, as the corpus' pie, not. + for (width, floored) in [("0", true), ("0.1", true), ("2", false)] { + let (_, report) = style(&qml( + &format!(r##""##), + "", + )); + assert_eq!(said(&report, "smaller than RustGIS draws them"), floored, "{width}: {:?}", report.notes); + } + // Between two values: the zero minimum radius is noted independently of the value range. let scaled = |lower: &str| { format!( r##""## @@ -2234,7 +2262,8 @@ fn diagrams_carry_over_as_pie_diagrams() { let (_, report) = style(&scaled("100")); assert!(said(&report, "from a lower value other than 0"), "{:?}", report.notes); let (_, report) = style(&scaled("0")); - assert!(report.notes.is_empty(), "{:?}", report.notes); + assert!(said(&report, "smaller than RustGIS draws them"), "{:?}", report.notes); + assert!(!said(&report, "from a lower value other than 0"), "{:?}", report.notes); // Several diagrams of a kind left out: only that is said. let two = r##""##; let (d, report) = style(&qml(two, "").replace(r#"diagramType="Pie""#, r#"diagramType="Histogram""#)); diff --git a/crates/map/src/raster.rs b/crates/map/src/raster.rs index 76efd7e254..38f792bf82 100644 --- a/crates/map/src/raster.rs +++ b/crates/map/src/raster.rs @@ -2169,12 +2169,30 @@ impl RasterLayer { self.masked_stats().is_some_and(|m| m.first().is_some_and(|s| s.count == 0)) } + /// Unique Values without an enumerated statistics list shows every configured value, just + /// as an explicit request for all classes does. None of those classes can be missed. + fn lists_every_class(&self, all_classes: bool) -> bool { + match &self.style.renderer { + RasterRenderer::UniqueValues { band, .. } => { + if all_classes { + return true; + } + let masked = self.masked_stats(); + let stats = masked.as_ref().and_then(|m| m.first()).or_else(|| self.data.as_ref()?.bands.get(*band).map(|b| &b.stats)); + stats.is_none_or(|s| s.categories.is_none()) + } + RasterRenderer::Classify { .. } => all_classes, + _ => false, + } + } + /// Whether the classes a legend lists for it ([`RasterLayer::legend_classes`]) come from a /// sample of its pixels: under a mask, a band of more than [`MASK_STATS_SAMPLE`] cells has its /// statistics read from every few cells ([`RasterMask::stats`]), so a class too rare for the /// sample can be missing (#1679). Not with `all_classes` for Unique Values or Classify, nor when - /// no class can be listed (a mask hiding every pixel included); a colour table's come from the sample only when its own statistics, - /// read whole, can't list its values. + /// no class can be listed (a mask hiding every pixel included). Unique Values without an + /// enumerated statistics list shows every class. A colour table's come from the sample only + /// when its own statistics, read whole, can't list its values. pub fn classes_from_sample(&self, all_classes: bool) -> bool { let Some(r) = &self.data else { return false }; if !self.style.renderer.draws_classes() || self.style.mask.is_empty() || self.classes_unknown() || self.mask_hides_all() { @@ -2184,7 +2202,7 @@ impl RasterLayer { let sampled = r.bands.get(band).is_some_and(|b| b.data.len() > MASK_STATS_SAMPLE); let from_sample = match &self.style.renderer { RasterRenderer::Colormap => r.bands.first().is_some_and(|b| b.stats.categories.is_none()), - _ => !all_classes, + _ => !self.lists_every_class(all_classes), }; sampled && from_sample } @@ -2194,10 +2212,11 @@ impl RasterLayer { /// while the map draws the full resolution: a class too rare for the overview can be missing /// (#1679). Not with `all_classes` for Unique Values or Classify, whose every class is listed /// whatever the statistics, nor when no class can be listed ([`RasterLayer::classes_unknown`], - /// or a mask hiding every pixel). + /// or a mask hiding every pixel), or when Unique Values lists every configured class because + /// its statistics have no enumerated values. pub fn classes_from_overview(&self, all_classes: bool) -> bool { let Some(r) = &self.data else { return false }; - let every = all_classes && !matches!(self.style.renderer, RasterRenderer::Colormap); + let every = self.lists_every_class(all_classes); self.style.renderer.draws_classes() && (self.in_place || r.overview.is_some()) && !every @@ -2472,6 +2491,35 @@ mod tests { assert_eq!(rl.legend_classes(true), None); } + /// Without an enumerated statistics list, Unique Values lists all configured classes; neither + /// an overview nor masked sampling can then hide a class from the legend. + #[test] + fn unique_values_listing_every_class_has_no_missing_class_warning() { + let side = 2049u32; + let gt = GeoTransform { origin_x: 0.0, origin_y: 1.0, pixel_width: 0.01, pixel_height: -0.01 }; + let cells = (0..side * side).map(|i| (i % 1000) as f32).collect(); + let raster = Raster::new("many", side, side, vec![cells], gt, Crs::LonLat, None, ColorHint::Gray, "UInt16").unwrap(); + assert!(raster.bands[0].stats.categories.is_none()); + let mut layer = RasterLayer::from_path(raster, "many.tif"); + layer.style.renderer = RasterRenderer::UniqueValues { + band: 0, + ramp: Ramp::Harbour, + values: (0..1000).map(|v| (f64::from(v), Rgba::rgb(30, 100, 200))).collect(), + }; + layer.in_place = true; + assert_eq!(layer.legend_classes(false).unwrap().len(), 1000); + assert!(!layer.classes_from_overview(false)); + layer.style.mask = RasterMask { values: vec![999.0], ..RasterMask::default() }; + assert_eq!(layer.legend_classes(false).unwrap().len(), 999); + assert!(!layer.classes_from_sample(false) && !layer.classes_from_overview(false)); + // Once a sampled subset supplies the legend, the rare-class warning applies again. + layer.style.mask = RasterMask { min: Some(0.0), max: Some(3.0), ..RasterMask::default() }; + // Sampling every second cell sees 0 and 2, missing the configured odd values. + assert_eq!(layer.legend_classes(false).unwrap().len(), 2); + assert!(layer.classes_from_sample(false) && layer.classes_from_overview(false)); + assert!(!layer.classes_from_sample(true) && !layer.classes_from_overview(true)); + } + /// A colour table has no entry for a negative value, and the map draws none: neither the /// Layers pane nor a legend lists it (#1679), where `as usize` would have taken entry 0. #[test] diff --git a/crates/map/src/source.rs b/crates/map/src/source.rs index 090a5083e1..c337e12901 100644 --- a/crates/map/src/source.rs +++ b/crates/map/src/source.rs @@ -1810,6 +1810,9 @@ mod tests { assert!(dataset.zm_dropped); assert_eq!(report.messages.iter().filter(|n| n.contains("conversion left out 2 source features")).count(), 1); assert_eq!(dataset.features.len(), 1); + let (_, counted) = layer.load_all_counted().unwrap(); + assert_eq!(counted.messages.iter().filter(|n| n.contains("conversion left out 2 source features")).count(), 1); + assert!(!counted.zm_noted, "a flag with no Z/M diagnostic cannot suppress the export warning"); } /// Project-restored disk and embedded date sources schedule their classification, share diff --git a/crates/map/src/tiles.rs b/crates/map/src/tiles.rs index 023a4578f3..0314e7aab1 100644 --- a/crates/map/src/tiles.rs +++ b/crates/map/src/tiles.rs @@ -222,7 +222,13 @@ pub fn checked_legends(legends: impl IntoIterator MAX_LEGEND_URL); + assert_eq!(login_address::advertised_len(&url), raw.len()); + assert_eq!(checked_legends([Legend { url: url.clone(), ..Default::default() }]).len(), 1); + let over = login_address::inherit(&document, &format!("{raw}x")); + assert!(checked_legends([Legend { url: over, ..Default::default() }]).is_empty()); + // Arbitrary escapes do not bypass the advertised byte limit. + let long = format!("{prefix}{}", "%41".repeat(MAX_LEGEND_URL)); + let long = login_address::inherit(&document, &long); + assert!(checked_legends([Legend { url: long, ..Default::default() }]).is_empty()); + let encoded = login_address::inner(&url).to_string(); + assert_eq!( + checked_legends([Legend { url: encoded, ..Default::default() }]).len(), + 1, + "saved URLs retain the same budget after unbinding" + ); + } + #[test] fn style_layers_are_checked() { assert!(checked_style_layers(r#"[{"id": "a", "type": "fill"}]"#).is_ok()); diff --git a/crates/plugins/src/panel.rs b/crates/plugins/src/panel.rs index 17dd76d602..9fb9f2d06b 100644 --- a/crates/plugins/src/panel.rs +++ b/crates/plugins/src/panel.rs @@ -1440,6 +1440,7 @@ mod tests { &d, &json!({"items": {"form": {"fields": [ {"id": "no choices", "kind": "select"}, + {"id": "no-choices", "kind": "select"}, {"id": "kind", "kind": "unknown"}, {"id": "pick", "kind": "select", "choices": ["a", "a", "b"]}, {"id": "pick", "kind": "text"}, diff --git a/crates/plugins/src/registry.rs b/crates/plugins/src/registry.rs index f0bcadc9a6..dd14ecb699 100644 --- a/crates/plugins/src/registry.rs +++ b/crates/plugins/src/registry.rs @@ -233,9 +233,8 @@ pub fn parse(text: &str, url: &str) -> Result, String> { parse_for(text, url, this_platform()) } -/// [`parse`], with a notice when packages by platform were skipped: one that doesn't check out -/// (no address, a bad SHA-256, an address that isn't allowed) costs its platform the plugin, so -/// it is said, with how many and which. None for a registry read whole. +/// [`parse`], with counted notices for skipped entries, invalid platform packages and entries +/// beyond the inspection limit. Names are samples when only some skipped ids can be read. pub fn parse_with_notice(text: &str, url: &str) -> Result<(Vec, Option), String> { parse_noting(text, url, this_platform()) } @@ -262,16 +261,13 @@ pub fn parse_noting(text: &str, url: &str, platform: Option<&str>) -> Result<(Ve let mut seen = std::collections::HashSet::new(); // The packages by platform that were skipped: how many, and the first few by name. let (mut skipped, mut which) = (0usize, Vec::new()); - // The plugins that were skipped whole (unreadable, or with no package that checks out): how - // many, how many of them have an id that can be shown, and the first few of those. - let (mut dropped, mut named, mut dropped_ids) = (0usize, 0usize, Vec::new()); + let omitted = list.len().saturating_sub(MAX_ENTRIES); + // The plugins skipped whole and the first few readable ids, in input order. + let (mut dropped, mut dropped_ids) = (0usize, Vec::new()); let mut drop_entry = |id: &str| { dropped += 1; - if valid_id(id, false) { - named += 1; - if dropped_ids.len() < 5 { - dropped_ids.push(id.to_string()); - } + if valid_id(id, false) && dropped_ids.len() < 5 { + dropped_ids.push(id.to_string()); } }; for v in list.into_iter().take(MAX_ENTRIES) { @@ -309,9 +305,16 @@ pub fn parse_noting(text: &str, url: &str, platform: Option<&str>) -> Result<(Ve } good }); + let shared_given = !e.package.is_empty() || !e.sha256.is_empty() || e.size > 0; let whole = !e.package.trim().is_empty() && is_sha256(&e.sha256); e.package = resolve(url, &e.package); let whole = whole && allowed_url(&e.package); + if shared_given && !whole && !e.packages.is_empty() { + skipped += 1; + if which.len() < 5 { + which.push(format!("{} (shared)", e.id)); + } + } match platform.and_then(|key| e.packages.get(key)) { Some(own) => (e.package, e.sha256, e.size) = (own.package.clone(), own.sha256.clone(), own.size), None if whole && !named => {} @@ -352,11 +355,20 @@ pub fn parse_noting(text: &str, url: &str, platform: Option<&str>) -> Result<(Ve } out.sort_by_key(|e| e.name.to_lowercase()); let mut notices = Vec::new(); + if omitted > 0 { + let entries = if omitted == 1 { "entry" } else { "entries" }; + let were = if omitted == 1 { "was" } else { "were" }; + notices.push(format!("{omitted} plugin registry {entries} past the first {MAX_ENTRIES} {were} not inspected")); + } if dropped > 0 { - let more = if named > dropped_ids.len() { ", …" } else { "" }; + let names = if dropped_ids.is_empty() { + String::new() + } else { + let more = if dropped > dropped_ids.len() { ", …" } else { "" }; + format!(": {}{more}", dropped_ids.join(", ")) + }; notices.push(format!( - "{dropped} plugin(s) in the plugin registry were skipped (an unreadable entry, id, version or minRustgisVersion, no package that checks out, or an id listed twice): {}{more}", - dropped_ids.join(", ") + "{dropped} plugin(s) in the plugin registry were skipped (an unreadable entry, id, version or minRustgisVersion, no package that checks out, or an id listed twice){names}" )); } if skipped > 0 { diff --git a/crates/plugins/src/settings.rs b/crates/plugins/src/settings.rs index 3ecd854f6d..da36db099f 100644 --- a/crates/plugins/src/settings.rs +++ b/crates/plugins/src/settings.rs @@ -51,8 +51,8 @@ pub const WRITE: &str = "host.setSettings"; /// Where settings are kept: one saved text per plugin id. trait Backing { - /// The text saved for `id`, or `None` when nothing was. At most one byte past [`MAX_SAVED`] - /// is read from where it is kept (too much is told by [`parse`]). + /// The text saved for `id`, or `None` when nothing was. Filesystem reads stop one byte past + /// [`MAX_SAVED`]; browser storage already holds the whole string. [`parse`] checks both. fn load(&self, id: &str) -> Result, String>; /// Replaces the text saved for `id`; a failure leaves what was saved as it was. fn save(&self, id: &str, text: &str) -> Result<(), String>; diff --git a/crates/plugins/src/store.rs b/crates/plugins/src/store.rs index 0c88f0bc4f..cfc85ca541 100644 --- a/crates/plugins/src/store.rs +++ b/crates/plugins/src/store.rs @@ -387,10 +387,20 @@ pub fn uninstall(id: &str) -> Result<(), String> { None => e, }; if is_package { - if let Some(unpacked) = unpacked_dir(id).filter(|u| u.is_dir()) { - std::fs::remove_dir_all(&unpacked).map_err(|e| kept(format!("{}: {e}", unpacked.display())))?; + let mut failures = Vec::new(); + if let Some(unpacked) = unpacked_dir(id).filter(|u| u.is_dir()) + && let Err(e) = std::fs::remove_dir_all(&unpacked) + { + failures.push(format!("{}: {e}", unpacked.display())); + } + // The package is already gone: attempt independent cleanup even when one directory + // cannot be removed, and name every leftover path before preserving its settings. + if let Err(e) = crate::pyenv::remove(id) { + failures.push(e); + } + if !failures.is_empty() { + return Err(kept(failures.join("; "))); } - crate::pyenv::remove(id).map_err(kept)?; } if erase { crate::settings::remove(id).map_err(kept)?; diff --git a/crates/plugins/src/tests.rs b/crates/plugins/src/tests.rs index 4398eb51e8..24a5bd0594 100644 --- a/crates/plugins/src/tests.rs +++ b/crates/plugins/src/tests.rs @@ -458,6 +458,29 @@ fn pinned_tool_schema_refuses_changed_defaults_display_and_stale_load() { crate::unload("unit-schema-pin"); } +/// Registry scan limits and omitted identifier samples remain explicit. +#[test] +fn registry_limits_and_unnamed_skips_are_counted_without_a_dangling_list() { + let entry = json!({"id":"kept", "name":"Kept", "version":"1.0.0", "package":"kept.rgplugin", "sha256":"a".repeat(64)}); + let url = "https://plugins.rustgis.app/registry.json"; + let (kept, notice) = registry::parse_with_notice(&json!([entry.clone()]).to_string(), url).unwrap(); + assert_eq!(kept.len(), 1); + assert_eq!(notice, None); + let (kept, notice) = registry::parse_with_notice("[7,8]", url).unwrap(); + assert!(kept.is_empty()); + let notice = notice.unwrap(); + assert!(notice.starts_with("2 plugin(s)") && notice.ends_with("listed twice)"), "{notice}"); + // Cheap malformed entries fill the inspected window; the valid tail is deliberately + // uninspected, rather than silently disappearing or being counted as malformed. + let mut entries = vec![Value::Null; registry::MAX_ENTRIES]; + entries.push(entry); + let (kept, notice) = registry::parse_with_notice(&Value::Array(entries).to_string(), url).unwrap(); + assert!(kept.is_empty()); + let notice = notice.unwrap(); + assert!(notice.starts_with(&format!("1 plugin registry entry past the first {} was not inspected", registry::MAX_ENTRIES)), "{notice}"); + assert!(notice.contains(&format!("{} plugin(s) in the plugin registry were skipped", registry::MAX_ENTRIES)), "{notice}"); +} + /// Registry and blocklist parsing validates entries and resolves package addresses safely. #[test] fn registries_and_blocklists_are_read_defensively() { @@ -480,7 +503,7 @@ fn registries_and_blocklists_are_read_defensively() { let (_, notice) = registry::parse_with_notice(&text, "https://plugins.rustgis.app/registry.json").unwrap(); let notice = notice.unwrap_or_default(); assert!( - notice.starts_with("6 plugin(s) in the plugin registry were skipped") && notice.ends_with(": bad, nohash, bad-min, a-plugin"), + notice.starts_with("6 plugin(s) in the plugin registry were skipped") && notice.ends_with(": bad, nohash, bad-min, a-plugin, …"), "{notice}" ); assert_eq!(v[1].package, "https://plugins.rustgis.app/packages/b/b-1.2.0.rgplugin"); @@ -1986,11 +2009,15 @@ fn process_packages_unpack_safely() { { use std::os::unix::fs::PermissionsExt; crate::settings::patch("unpacked", &json!({"values": {"kept": true}})).unwrap(); + let environment = crate::pyenv::env_dir("unpacked").unwrap(); + std::fs::create_dir_all(&environment).unwrap(); + std::fs::write(environment.join("leftover.txt"), "synthetic environment").unwrap(); let parent = plugins.join("unpacked"); let mode = std::fs::metadata(&parent).unwrap().permissions(); std::fs::set_permissions(&parent, std::fs::Permissions::from_mode(0o555)).unwrap(); let failed = crate::store::uninstall("unpacked"); std::fs::set_permissions(&parent, mode).unwrap(); + assert!(!environment.exists(), "independent Python cleanup must run after an unpacked-folder failure"); // Root removes it anyway: then there is nothing to check here. if let Err(e) = failed { assert!(e.contains("its saved settings were kept") && e.contains(&settings.display().to_string()), "{e}"); @@ -2656,11 +2683,23 @@ fn a_bad_platform_package_is_dropped_alone() { let notice = notice.unwrap(); // A plugin left with no package at all is said as a plugin skipped, first. assert!(notice.starts_with("1 plugin(s) in the plugin registry were skipped") && notice.contains(": g-no-address; "), "{notice}"); - assert!(notice.contains("; 6 package(s) for a platform in the plugin registry were skipped"), "{notice}"); + assert!(notice.contains("; 7 package(s) for a platform in the plugin registry were skipped"), "{notice}"); assert!( notice.ends_with(": a-no-address (Linux), b-no-hash (Linux), c-not-an-object (Linux), d-bad-size (Linux), e-own-too (Linux), …"), "{notice}" ); + // An invalid shared fallback beside good platform packages is counted on every platform. + let fallback = json!([{"id": "shared", "name": "Shared", "version": "1.0.0", + "package": "http://insecure.example.org/a.rgplugin", "sha256": any, + "packages": {"windows": good}}]) + .to_string(); + for platform in ["windows", "linux", "macos"] { + let (entries, notice) = registry::parse_noting(&fallback, url, Some(platform)).unwrap(); + assert_eq!(entries.len(), 1); + let notice = notice.unwrap(); + assert!(notice.contains("1 package(s)") && notice.ends_with("shared (shared)"), "{platform}: {notice}"); + assert_eq!(entries[0].available(), platform == "windows"); + } let clean = json!([ {"id": "ok", "name": "ok", "version": "1.0.0", "packages": {"windows": good, "plan9": {"package": "p.rgplugin", "sha256": lin}}}, // `null` is no packages: the entry's own package, as without the field. diff --git a/crates/tiles/src/address.rs b/crates/tiles/src/address.rs index 020c38566f..db24c8bb39 100644 --- a/crates/tiles/src/address.rs +++ b/crates/tiles/src/address.rs @@ -60,8 +60,9 @@ impl Origin { Ok(self) } - /// Retains this trusted origin's address while narrowing its observed reach. - /// This constructor is for in-process transport/UI code, never document deserialization. + /// Records the address and reach observed by trusted in-process transport/UI code. The + /// observed peer can be local even when the address's spelling looks public; this does not + /// narrow a previous origin. Never use document data as the observed reach. pub fn with_observed_reach(url: &str, reach: Reach) -> Result { check_reach(url, Reach::Local)?; if reach == Reach::Forbidden { diff --git a/crates/tiles/src/geocode.rs b/crates/tiles/src/geocode.rs index e259b84288..b2364bb20b 100644 --- a/crates/tiles/src/geocode.rs +++ b/crates/tiles/src/geocode.rs @@ -255,6 +255,8 @@ impl Geocoder { found.limited = entries.len().saturating_sub(i); break; } + // An overlong label is a malformed Census response, not a candidate with no usable + // position: refuse it whole rather than shorten or substitute its matched address. if self.kind == Kind::Census && e.get("matchedAddress").and_then(Value::as_str).is_some_and(|name| name.chars().nth(500).is_some()) { diff --git a/crates/tiles/src/pmtiles.rs b/crates/tiles/src/pmtiles.rs index b454cee0fa..88ed3d587d 100644 --- a/crates/tiles/src/pmtiles.rs +++ b/crates/tiles/src/pmtiles.rs @@ -481,7 +481,10 @@ pub fn info(archive: &str) -> Result>, String> { pub fn info_blocking(archive: &str, timeout: std::time::Duration) -> Result, String> { let (tx, rx) = std::sync::mpsc::channel(); with_info(archive, Box::new(move |r| tx.send(r).unwrap_or_default())); - rx.recv_timeout(timeout).map_err(|_| format!("{archive}: no answer in {} s", timeout.as_secs()))? + rx.recv_timeout(timeout).map_err(|error| match error { + std::sync::mpsc::RecvTimeoutError::Timeout => format!("{archive}: no answer in {} s", timeout.as_secs()), + std::sync::mpsc::RecvTimeoutError::Disconnected => format!("{archive}: metadata read ended without an answer"), + })? } /// Calls `f` with the archive's info, once it is read. @@ -496,9 +499,17 @@ fn with_info(archive: &str, f: Box, String>) + Send> return; } let epoch = crate::cache::archive_epoch(archive); + with_info_at_epoch(archive, epoch, f); +} + +/// Admits a caller already checked and bound to its lifetime. Reload may retire the captured +/// epoch before admission; that caller still gets exactly one terminal answer. +fn with_info_at_epoch(archive: &str, epoch: u64, f: Box, String>) + Send>) { + let mut f = Some(f); let result = crate::cache::if_archive_current(archive, epoch, || { + let f = f.take()?; let mut all = archives().lock().unwrap_or_else(|e| e.into_inner()); - match all.get_mut(archive) { + Some(match all.get_mut(archive) { Some(State::Ready(i)) => (Some((f, Ok(i.clone()))), false, None), Some(State::Loading(waiting)) => { waiting.push(f); @@ -511,8 +522,9 @@ fn with_info(archive: &str, f: Box, String>) + Send> all.insert(archive.to_string(), State::Loading(vec![f])); (None, true, None) } - } - }); + }) + }) + .flatten(); if let Some((answer, start, retry_at)) = result { if let Some(at) = retry_at { crate::cache::notify_at(at); @@ -524,6 +536,9 @@ fn with_info(archive: &str, f: Box, String>) + Send> load(archive.to_string(), epoch); } } + if let Some(f) = f { + f(Err(crate::cache::DROPPED.into())); + } } /// Reads an archive's header, root directory and metadata, then answers whoever waits. @@ -1293,6 +1308,28 @@ pub(crate) mod tests { assert!(failures.is_empty(), "{failures:?}"); } + #[test] + fn a_reload_before_metadata_admission_answers_the_old_caller() { + let _turn = crate::cache::turn(); + let source = "/pmtiles-stale-admission-regression.pmtiles"; + let old_epoch = crate::cache::archive_epoch(source); + crate::cache::reload_archive(source); + let (tx, rx) = std::sync::mpsc::channel(); + with_info_at_epoch(source, old_epoch, Box::new(move |result| tx.send(result.map(|info| info.attribution())).unwrap())); + assert_eq!(rx.try_recv().unwrap(), Err(crate::cache::DROPPED.to_string())); + assert!(matches!(rx.try_recv(), Err(std::sync::mpsc::TryRecvError::Disconnected)), "one terminal callback"); + assert!(!archives().lock().unwrap().contains_key(source), "stale admission publishes no metadata"); + // The same caller admitted at the current epoch receives the ready archive normally. + let bytes = archive(&[(0, 0, 0, vec![1])], 1, true, "{}"); + let (ready, _, _) = parse_head(&bytes).unwrap(); + archives().lock().unwrap().insert(source.into(), State::Ready(Arc::new(ready))); + let (tx, rx) = std::sync::mpsc::channel(); + with_info_at_epoch(source, crate::cache::archive_epoch(source), Box::new(move |result| tx.send(result.is_ok()).unwrap())); + assert!(rx.try_recv().unwrap()); + assert!(matches!(rx.try_recv(), Err(std::sync::mpsc::TryRecvError::Disconnected))); + archives().lock().unwrap().remove(source); + } + #[test] fn reloaded_archive_rejects_old_metadata_and_leaf_answers() { let _turn = crate::cache::turn(); diff --git a/crates/ui-egui/src/assistantpane.rs b/crates/ui-egui/src/assistantpane.rs index 520bf03a2e..e386da7af3 100644 --- a/crates/ui-egui/src/assistantpane.rs +++ b/crates/ui-egui/src/assistantpane.rs @@ -76,6 +76,9 @@ pub struct AssistantState { /// may block or ask (desktop only). #[cfg(not(target_arch = "wasm32"))] sign_in_read: Option, + /// A refused saved-account store is reported once through this app session's notices. + #[cfg(not(target_arch = "wasm32"))] + sign_in_store_error_said: bool, /// The first sign-in's confirmation that the ChatGPT plan is in use, until dismissed. welcome: bool, /// A failed request to send again: its generation and when. @@ -260,19 +263,21 @@ fn sync_skills(app: &mut RustGisApp, force: bool) { } } -/// A sign-in read under way: the account it checks (`None`: take whichever is selected) and -/// where the worker leaves what it read. +/// A saved account and the reason its store or tokens could not be read. +#[cfg(not(target_arch = "wasm32"))] +type SignInResult = (Option, Option); +/// A read under way: the account checked (`None`: whichever is selected) and its result slot. #[cfg(not(target_arch = "wasm32"))] -type SignInRead = (Option, Slot>); +type SignInRead = (Option, Slot); /// Starts reading the selected ChatGPT sign-in on a worker; [`poll_sign_in`] applies it. With /// `checking`, it only says whether that sign-in is still there (after a failed request). #[cfg(not(target_arch = "wasm32"))] fn read_sign_in(app: &mut RustGisApp, checking: Option, ctx: &egui::Context) { - let slot: Slot> = Slot::default(); + let slot: Slot = Slot::default(); let (out, ctx) = (slot.clone(), ctx.clone()); std::thread::spawn(move || { - let c = chatgpt::load_any(); + let c = chatgpt::load_any_report(); if let Ok(mut s) = out.lock() { *s = Some(c); } @@ -281,6 +286,28 @@ fn read_sign_in(app: &mut RustGisApp, checking: Option, ctx: & app.assistant.sign_in_read = Some((checking, slot)); } +/// Keeps account-store refusal visible without emitting the same failure on every poll. +#[cfg(not(target_arch = "wasm32"))] +fn sign_in_store_error(app: &mut RustGisApp, error: rustgis_assistant::Error) { + if !app.assistant.sign_in_store_error_said { + app.assistant.sign_in_store_error_said = true; + app.session.warn(crate::i18n::fmt("Could not read saved ChatGPT sign-ins: {reason}", &[("reason", &error.to_string())])); + } + app.assistant.login_error = Some(error.auth_context()); +} + +/// Reads the account list while preserving the same refusal notice as the background read. +#[cfg(not(target_arch = "wasm32"))] +fn saved_accounts(app: &mut RustGisApp) -> Vec { + match chatgpt::accounts() { + Ok(accounts) => accounts, + Err(error) => { + sign_in_store_error(app, error); + Vec::new() + } + } +} + /// Applies a finished sign-in read, if the sign-in provider is still the one chosen (account /// actions drop a read begun before them). A check after a failed request never takes another /// account; when its sign-in is gone (expired, revoked, signed out elsewhere) the key is cleared @@ -293,6 +320,10 @@ fn poll_sign_in(app: &mut RustGisApp, ctx: &egui::Context) { let checking = checking.clone(); app.assistant.sign_in_read = None; let Some(cfg) = app.assistant.agent.as_ref().map(|a| a.config.clone()).filter(Config::login) else { return }; + let (read, warning) = read; + if let Some(warning) = warning { + sign_in_store_error(app, warning); + } let usable = read.filter(|c| c.signed_in() && c.plan_allowed()); if let Some(binding) = checking { // Still signed in, or another account selected and signed in (requests for this one @@ -490,7 +521,7 @@ fn signed_in(app: &mut RustGisApp, c: &chatgpt::Credentials, ctx: &egui::Context app.assistant.sign_in_read = None; app.assistant.account = Some(c.account()); app.assistant.login_error = None; - let saved = chatgpt::accounts().unwrap_or_default(); + let saved = saved_accounts(app); // "Use another account" with an account already saved registers RustGIS for it again. app.assistant.duplicate = saved.iter().any(|a| a.subject == c.subject && a.client_id != c.client_id); app.assistant.accounts = Some(saved); @@ -1110,7 +1141,10 @@ fn account_row(app: &mut RustGisApp, ui: &mut Ui, t: &Tokens, cfg: &mut Config) } return; } - let saved = app.assistant.accounts.get_or_insert_with(|| chatgpt::accounts().unwrap_or_default()).clone(); + if app.assistant.accounts.is_none() { + app.assistant.accounts = Some(saved_accounts(app)); + } + let saved = app.assistant.accounts.clone().unwrap_or_default(); let active = saved.iter().find(|c| c.client_id == cfg.client_id && c.subject == cfg.account); let label = active.map(chatgpt::Credentials::label).unwrap_or_else(|| tl!("Choose an account").into()); let picker = egui::ComboBox::from_id_salt("assistant_account").selected_text(label).width(180.0).truncate().show_ui(ui, |ui| { @@ -2604,19 +2638,48 @@ pub(crate) mod tests { } h.state_mut().assistant.models_slot = None; // A read begun before choosing an account must not undo the choice when it finishes. - let stale: Slot> = Slot::default(); + let stale: Slot = Slot::default(); h.state_mut().assistant.sign_in_read = Some((None, stale.clone())); h.get_by_value(&a.label()).click(); h.run_steps(4); h.get_by_label(&b.label()).click(); h.run_steps(4); - *stale.lock().unwrap() = Some(Some(a.clone())); + *stale.lock().unwrap() = Some((Some(a.clone()), None)); h.run_steps(4); assert!(h.state().assistant.models_slot.is_some()); assert_eq!(h.state().assistant.agent.as_ref().unwrap().config.client_id, b.client_id); assert_eq!(chatgpt::load_any().unwrap().client_id, b.client_id); } + /// A refused store is distinct from no saved account and reaches notices once, not just logs. + #[cfg(not(target_arch = "wasm32"))] + #[test] + fn a_refused_sign_in_read_is_visible_once_and_a_clean_read_is_quiet() { + let ctx = egui::Context::default(); + let mut app = plan_app(); + app.session.notices.clear(); + let finish = |app: &mut RustGisApp, warning: Option| { + let slot: Slot = Slot::default(); + *slot.lock().unwrap() = Some((None, warning)); + app.assistant.sign_in_read = Some((None, slot)); + poll_sign_in(app, &ctx); + }; + finish(&mut app, None); + assert!(app.session.notices.is_empty()); + let warning = + rustgis_assistant::Error::new(rustgis_assistant::Recovery::Auth, "the ChatGPT sign-ins were saved by a newer RustGIS"); + finish(&mut app, Some(warning.clone())); + finish(&mut app, Some(warning)); + assert_eq!(app.session.notices.len(), 1); + assert!(app.session.notices[0].text.contains("newer RustGIS")); + assert!(app.assistant.login_error.as_ref().unwrap().detail.contains("newer RustGIS")); + let mut other = plan_app(); + other.assistant.agent.as_mut().unwrap().config = Config::from_env("custom"); + other.session.notices.clear(); + finish(&mut other, Some("a stale read for another provider".into())); + assert!(other.session.notices.is_empty() && other.assistant.login_error.is_none()); + } + #[cfg(not(target_arch = "wasm32"))] #[test] fn a_sign_in_check_never_switches_accounts_and_a_switch_stops_work() { @@ -2633,8 +2696,8 @@ pub(crate) mod tests { ..Default::default() }; let finish = |app: &mut RustGisApp, checking: Option, read: Option| { - let slot: Slot> = Slot::default(); - *slot.lock().unwrap() = Some(read); + let slot: Slot = Slot::default(); + *slot.lock().unwrap() = Some((read, None)); app.assistant.sign_in_read = Some((checking, slot)); poll_sign_in(app, &ctx); }; diff --git a/crates/ui-egui/src/lib.rs b/crates/ui-egui/src/lib.rs index e0f4037ca5..d2baea6aa3 100644 --- a/crates/ui-egui/src/lib.rs +++ b/crates/ui-egui/src/lib.rs @@ -371,6 +371,9 @@ pub struct Services { /// Opens a file picker for `"data"`, `"project"`, or `"shapefile"`; results land in `inbox`. /// Shapefile picks carry [`SHAPEFILE_PICK`] after sidecars are gathered into ZIPs. pub open_async: Option>, + /// Snapshot the pending options immediately before a picker is submitted. Phone shells + /// retain these only on admission, so a refused picker cannot overwrite an active request. + pub open_options: Option>, /// Asks the device's camera for a photo (a phone's shell): the photo arrives in /// [`RustGisApp::photos`]. `None` where there is no camera to ask. pub take_photo: Option Result<(), String>>>, @@ -3716,6 +3719,9 @@ impl RustGisApp { self.ui.pending_open = None; // `null`: the dialog, for the file picked. self.ui.pending_csv = Some(opts.unwrap_or(Value::Null)); + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("text"); } else if let Some(path) = dialogs::pick_open( &["csv", "tsv", "txt", "dat", "gz", "xlsx", "xlsm", "xlsb", "xls", "ods"], @@ -3750,6 +3756,9 @@ impl RustGisApp { self.ui.next_dxf = self.ui.next_dxf.checked_add(1).ok_or(tl!("DXF picker request limit reached"))?; let id = self.ui.next_dxf; self.ui.pending_dxf.insert(id, opts.unwrap_or(Value::Null)); + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open(&format!("dxf:{id}")); } else if let Some(path) = dialogs::pick_open(&["dxf"], tl!("DXF drawing")) { return match opts { @@ -3907,6 +3916,9 @@ impl RustGisApp { self.ui.pending_open = None; self.ui.pending_csv = None; self.ui.pending_shapefile = Some(json!({"crs": text})); + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("shapefile"); } else if let Some(path) = dialogs::pick_open(&["shp", "zip"], tl!("Shapefile")) { // Read on a thread of its own, as Add Data's other files are. @@ -3923,6 +3935,9 @@ impl RustGisApp { self.ui.pending_picture = None; self.ui.pending_open = None; self.ui.pending_csv = None; + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("data"); } else if let Some(path) = dialogs::pick_open( &[ @@ -3991,6 +4006,9 @@ impl RustGisApp { self.ui.pending_layout_picture = false; self.ui.pending_picture = None; self.ui.pending_open = None; + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("project"); self.ui.backstage = false; } else if let Some(path) = dialogs::pick_open( @@ -4081,6 +4099,9 @@ impl RustGisApp { "ui.reportLayoutFile" => { if let Some(open) = &self.services.open_async { self.ui.pending_open = Some("report.existingTemplate"); + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("item"); } else if let Some(path) = dialogs::pick_open(&["rgislayout"], tl!("RustGIS layout")) { return self.run("report.existingTemplate", json!({"path": path})); @@ -4125,6 +4146,9 @@ impl RustGisApp { self.ui.pending_layout_picture = false; self.ui.pending_picture = None; self.ui.pending_open = None; + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("layout"); } else if let Some(path) = dialogs::pick_open(&["rgislayout"], tl!("RustGIS layout")) { return self.run("layout.import", json!({"path": path})); @@ -4136,6 +4160,9 @@ impl RustGisApp { self.ui.pending_layout_picture = true; self.ui.pending_picture = None; self.ui.pending_open = None; + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("picture"); } else if let Some(path) = dialogs::pick_open(&["png", "jpg", "jpeg"], tl!("Pictures")) { let v = self.run("layout.addElement", json!({"layout": li, "kind": "picture", "path": path}))?; @@ -4212,6 +4239,9 @@ impl RustGisApp { }; if let Some(open) = &self.services.open_async { self.ui.pending_open = Some(cmd); + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open(match kind { "ramps" => "ramp", "geolibreProject" => "project", @@ -4421,6 +4451,9 @@ impl RustGisApp { self.ui.pending_layout_picture = false; self.ui.pending_picture = None; self.ui.pending_open = Some("presentation.imagePage"); + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("picture"); } else if let Some(path) = dialogs::pick_open(&["png", "jpg", "jpeg"], tl!("Pictures")) { return self.run("presentation.imagePage", json!({"path": path})); @@ -4532,6 +4565,9 @@ impl RustGisApp { let cmd = if p.get("map").and_then(Value::as_bool) == Some(true) { "map.thumbnail" } else { "project.thumbnail" }; if let Some(open) = &self.services.open_async { self.ui.pending_open = Some(cmd); + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("picture"); } else if let Some(path) = dialogs::pick_open(&["png", "jpg", "jpeg"], tl!("Pictures")) { return self.run(cmd, json!({"path": path})); @@ -4639,7 +4675,8 @@ impl RustGisApp { let name = format!("{}.{ext}", l.name); // The features written, read whole, give their own extent (not a raster's, or the // one a streamed layer's source declares). - let notes = rustgis_engine::data::export_notes([(ds, ds.extent())], &name, crs); + let mut notes = rustgis_engine::data::export_notes([(ds, ds.extent())], &name, crs); + rustgis_engine::cmd::share::coalesce_read_zm(&mut notes, &name, ds, &lost); let mut dbf_notes = Vec::new(); if ext == "zip" { dbf_notes.extend(io::shapefile_names_note(&ds.fields, &name)); @@ -4654,7 +4691,7 @@ impl RustGisApp { if lost.rows > 0 { dbf_notes.push(format!("{name}: {} rows couldn't be read and were left out", lost.rows)); } - dbf_notes.extend(rustgis_engine::cmd::share::read_date_notes(&name, &l.name, &lost)); + dbf_notes.extend(rustgis_engine::cmd::share::read_notes(&name, &l.name, &lost)); // CSV holds a point a row, GPX a name for each feature and no areas. dbf_notes.extend(io::export_losses_notes(ds, &ext, io::name_field(ds), &name)); if let Some(note) = io::left_out_note(&name, &ext, left_out) { @@ -4943,6 +4980,9 @@ impl RustGisApp { if let Some(open) = &self.services.open_async { self.ui.pending_layout_picture = false; self.ui.pending_open = None; + if let Some(snapshot) = &self.services.open_options { + snapshot(&self.ui); + } open("picture"); } else if let Some(path) = dialogs::pick_open(&["png", "jpg", "jpeg"], tl!("Pictures")) { return self.add_picture(json!({"path": path})); diff --git a/crates/ui-egui/tests/picker_admission.rs b/crates/ui-egui/tests/picker_admission.rs new file mode 100644 index 0000000000..437ec0d6b4 --- /dev/null +++ b/crates/ui-egui/tests/picker_admission.rs @@ -0,0 +1,27 @@ +//! Admission snapshots are taken after options are prepared, before the platform is called. +use rustgis_ui_egui::{RustGisApp, Services}; +use serde_json::{Value, json}; +use std::sync::{Arc, Mutex}; + +#[test] +fn two_requests_in_one_frame_keep_the_first_admitted_options() { + let candidate = Arc::new(Mutex::new(None::)); + let admitted = Arc::new(Mutex::new(None::)); + let snapshots = candidate.clone(); + let (pending, accepted) = (candidate.clone(), admitted.clone()); + let mut app = RustGisApp::with_sample().with_services(Services { + open_options: Some(Box::new(move |ui| *snapshots.lock().unwrap() = ui.pending_csv.clone())), + open_async: Some(Box::new(move |_| { + let mut first = accepted.lock().unwrap(); + if first.is_none() { + *first = pending.lock().unwrap().clone(); + } + })), + ..Default::default() + }); + let options = json!({"x": "longitude", "y": "latitude", "delimiter": ";"}); + app.run("ui.addDelimited", options.clone()).unwrap(); + app.run("ui.addData", Value::Null).unwrap(); + assert!(candidate.lock().unwrap().is_none()); + assert_eq!(*admitted.lock().unwrap(), Some(options)); +} diff --git a/crates/ui-touch/src/copies.rs b/crates/ui-touch/src/copies.rs index 09319d810f..6bffdcc66f 100644 --- a/crates/ui-touch/src/copies.rs +++ b/crates/ui-touch/src/copies.rs @@ -10,14 +10,14 @@ //! //! A pick still being copied into is kept too. The phones mark a pick's folder with [`COPYING`] //! while they copy into it, and write each copy under a `.part` name of its own, renamed once -//! whole. A mark or a `.part` file left by a copy the app was killed during stops counting after -//! [`STALE`], and its pick is judged like any other. +//! whole. A mark owned by this process protects the whole batch regardless of its age. A mark +//! from a previous process, or a `.part` left after a crash, stops counting after [`STALE`]. use rustgis_ui_egui::rustgis_engine::data::dirs; use rustgis_ui_egui::{RustGisApp, i18n, tl}; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, Ordering}; -use std::sync::{Arc, Mutex}; +use std::sync::{Arc, Mutex, OnceLock}; /// The deepest a file of a pick is (`picked///`), looked for a level /// deeper: a pick holds no deeper folders, and a link out of it is never followed. @@ -27,7 +27,62 @@ const DEPTH: usize = 3; /// pick, and remove once done. pub const COPYING: &str = ".copying"; -/// How long a pick's [`COPYING`] mark or `.part` file counts after it was last written: a copy +/// The process lifetime token shared with the platform copy worker. Android installs Java's +/// static UUID before the phone is used; iOS uses the token generated here. +static COPY_OWNER: OnceLock = OnceLock::new(); + +/// This process's document-copy token. It is independent of the wall clock and process-id reuse. +pub fn copy_owner() -> &'static str { + use std::hash::BuildHasher as _; + COPY_OWNER.get_or_init(|| { + let random = std::collections::hash_map::RandomState::new(); + format!("process-{:016x}{:016x}", random.hash_one(0u8), random.hash_one(1u8)) + }) +} + +/// Installs Android's process-wide Java token before the front end can inspect unused copies. +/// A conflicting second installation is refused rather than treating another owner as active. +pub fn set_copy_owner(owner: String) -> Result<(), String> { + if owner.is_empty() || owner.len() > 128 || !owner.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'-') { + return Err(tl!("The document-copy owner is invalid.").to_string()); + } + if let Some(current) = COPY_OWNER.get() { + return if current == &owner { Ok(()) } else { Err(tl!("The document-copy owner was already initialized.").to_string()) }; + } + COPY_OWNER.set(owner).map_err(|_| tl!("The document-copy owner was already initialized.").to_string()) +} + +/// A document batch's mark, removed on completion or unwinding. The platform keeps this guard +/// alive across document-provider waits and gaps between files. +pub struct Copying { + mark: PathBuf, +} + +/// Marks `folder` for this process for the guard's whole lifetime. +pub fn mark_copying(folder: &Path) -> std::io::Result { + std::fs::create_dir_all(folder)?; + let mark = folder.join(COPYING); + std::fs::write(&mark, copy_owner())?; + Ok(Copying { mark }) +} + +impl Drop for Copying { + fn drop(&mut self) { + if owned_mark(&self.mark) { + let _ = std::fs::remove_file(&self.mark); + } + } +} + +/// Reads at most the token limit plus one byte, even for a damaged or hostile marker file. +fn owned_mark(file: &Path) -> bool { + use std::io::Read as _; + let Ok(file) = std::fs::File::open(file) else { return false }; + let mut owner = Vec::with_capacity(129); + file.take(129).read_to_end(&mut owner).is_ok() && owner == copy_owner().as_bytes() +} + +/// How long an abandoned or legacy [`COPYING`] mark or `.part` file counts after it was last written: a copy /// writes its `.part` as it goes, but iOS may first download a document from iCloud for a while. const STALE: std::time::Duration = std::time::Duration::from_secs(60 * 60); @@ -133,11 +188,11 @@ fn gather(path: &Path, depth: usize, files: &mut Vec<(PathBuf, u64)>) -> bool { complete } -/// Whether `file` says its pick is being copied into: a [`COPYING`] mark or a `.part` copy, written -/// in the last [`STALE`]. +/// A mark owned by this process, however old; otherwise a legacy/abandoned mark or `.part` +/// copy written in the last [`STALE`]. fn in_flight(file: &Path) -> bool { - let marks = file.file_name().is_some_and(|n| n == COPYING) || file.extension().is_some_and(|e| e == "part"); - marks && recent(file) + let mark = file.file_name().is_some_and(|n| n == COPYING); + (mark && owned_mark(file)) || ((mark || file.extension().is_some_and(|e| e == "part")) && recent(file)) } /// Whether `file` was last written in the last [`STALE`] (or in the future, as a clock set back @@ -450,6 +505,34 @@ mod tests { std::fs::remove_dir_all(&root).unwrap(); } + #[test] + fn an_old_active_batch_between_documents_stays_protected_until_completion() { + let root = std::env::temp_dir().join(format!("rustgis-copies-owner-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let picked = root.join("picked"); + let pick = picked.join("7"); + std::fs::create_dir_all(pick.join("0")).unwrap(); + std::fs::write(pick.join("0/roads.gpkg"), b"whole").unwrap(); + let asked = unused(&picked, "{}", &AtomicBool::new(false)).unwrap(); + let guard = mark_copying(&pick).unwrap(); + let mark = pick.join(COPYING); + let long_ago = std::time::SystemTime::now() - STALE - std::time::Duration::from_secs(60); + std::fs::File::options().write(true).open(&mark).unwrap().set_modified(long_ago).unwrap(); + // No .part exists: a provider is paused before the next document begins. + let found = unused(&picked, "{}", &AtomicBool::new(false)).unwrap(); + assert!(found.picks.is_empty()); + assert_eq!(found.busy_picks.as_slice(), std::slice::from_ref(&pick)); + assert_eq!(remove_unused(&picked, &asked, "{}", &AtomicBool::new(false)).unwrap().busy, 1); + assert!(pick.join("0/roads.gpkg").exists()); + drop(guard); + assert!(!mark.exists()); + assert_eq!(unused(&picked, "{}", &AtomicBool::new(false)).unwrap().picks.as_slice(), std::slice::from_ref(&pick)); + std::fs::write(&mark, b"previous-process-owner").unwrap(); + std::fs::File::options().write(true).open(&mark).unwrap().set_modified(long_ago).unwrap(); + assert_eq!(unused(&picked, "{}", &AtomicBool::new(false)).unwrap().picks, [pick]); + std::fs::remove_dir_all(&root).unwrap(); + } + #[test] fn a_pick_left_by_a_copy_cut_short_long_ago_is_judged_like_any_other() { let root = std::env::temp_dir().join(format!("rustgis-copies-stale-{}", std::process::id())); diff --git a/crates/ui-touch/src/lib.rs b/crates/ui-touch/src/lib.rs index 13a3617777..c29d7b4d56 100644 --- a/crates/ui-touch/src/lib.rs +++ b/crates/ui-touch/src/lib.rs @@ -68,7 +68,7 @@ fn set_sheet(ctx: &egui::Context, sheet: Sheet) { ctx.data_mut(|d| d.insert_temp(egui::Id::unique("touch_sheet"), sheet)); } -pub use copies::COPYING; +pub use copies::{COPYING, Copying, copy_owner, mark_copying, set_copy_owner}; /// Whether unused copies of picked files are being removed (Remove Unused Copies): a phone's shell /// holds files sent to the app until it is done, as one could open a project naming a copy the diff --git a/crates/ui-touch/src/track.rs b/crates/ui-touch/src/track.rs index 045132e00b..89f2cca76c 100644 --- a/crates/ui-touch/src/track.rs +++ b/crates/ui-touch/src/track.rs @@ -136,8 +136,6 @@ pub(crate) struct Average { pub lat: f64, /// How far the positions lie from it, in metres (their root mean square distance). pub spread: f64, - /// The mean of the accuracies they gave, in metres, when every one gave one. - pub accuracy: Option, pub count: usize, } @@ -163,8 +161,7 @@ pub(crate) fn average(fixes: &[Fix]) -> Option { dx * dx + dy * dy }; let spread = (fixes.iter().map(metres).sum::() / fixes.len() as f64).sqrt(); - let accuracy = fixes.iter().map(|f| f.accuracy).sum::>().map(|sum| sum / fixes.len() as f64); - Some(Average { lon: (lon + 540.0).rem_euclid(360.0) - 180.0, lat, spread, accuracy, count: fixes.len() }) + Some(Average { lon: (lon + 540.0).rem_euclid(360.0) - 180.0, lat, spread, count: fixes.len() }) } /// What the Capture sheet says of an average being taken. @@ -332,15 +329,14 @@ mod tests { let a = average(&[fix(10.0, 20.0, Some(5.0), None), fix(10.0002, 20.0, Some(5.0), None)]).unwrap(); assert!((a.lon - 10.0001).abs() < 1e-9 && (a.lat - 20.0).abs() < 1e-9, "{a:?}"); assert_eq!(a.count, 2); - assert_eq!(a.accuracy, Some(5.0)); // About 21 m apart: each about 10.5 m from the middle. assert!((a.spread - 10.46).abs() < 0.2, "{a:?}"); // One three times as exact counts nine times as much. let b = average(&[fix(10.0, 20.0, Some(1.0), None), fix(10.001, 20.0, Some(3.0), None)]).unwrap(); assert!((b.lon - 10.0001).abs() < 1e-9, "{b:?}"); - // One without an accuracy: all count the same, and no accuracy is given. + // One without an accuracy: all count the same. let c = average(&[fix(10.0, 20.0, Some(1.0), None), fix(10.001, 20.0, None, None)]).unwrap(); - assert!((c.lon - 10.0005).abs() < 1e-9 && c.accuracy.is_none(), "{c:?}"); + assert!((c.lon - 10.0005).abs() < 1e-9, "{c:?}"); // Accuracies out of all measure, or of nothing, still weigh: none is infinite or nothing. let e = average(&[fix(10.0, 20.0, Some(1e300), None), fix(10.001, 20.0, Some(1e300), None)]).unwrap(); assert!((e.lon - 10.0005).abs() < 1e-9, "{e:?}"); diff --git a/docs/privacy.md b/docs/privacy.md index 96eb48c827..5659c04cea 100644 --- a/docs/privacy.md +++ b/docs/privacy.md @@ -19,11 +19,14 @@ the AI Assistant are reachable from them. On top of that: positions nowhere, but the [AI Assistant](#ai-assistant-and-connected-agents), when you use it, can send a feature's geometry to the provider you configure if one of its tools reads it. - **Camera and photos**: a photo you take is saved in the app's storage and attached to the - feature you capture. It leaves the device only in a project or field package you save or send. + feature you capture. The original camera metadata is kept and may include GPS coordinates. + A field package created to bring work back from the phone copies its attached photos and + their metadata. A saved project or project package keeps the attached photo's path, without + copying the photo file. - **Files you open**: copied into the app's storage, where they are opened and edited. Remove - Unused Copies deletes the copies your project no longer uses. On iPhone and iPad these copies - are left out of the device's backups; if the system refuses that, the app says the copies - will be backed up. + Unused Copies deletes the copies your project no longer uses. Android app storage is excluded + from device backups. On iPhone and iPad these copies are left out of the device's backups; + if the system refuses that, the app says the copies will be backed up. - **Notifications** (Android): used only for the notice Android requires while a track is recorded with the app out of sight. @@ -54,15 +57,18 @@ token or signature does not hide the HTTP status or other explanation. The compl fragment are omitted, including values with unescaped spaces. For an unknown URL embedded in free text, a secret-named query or fragment still omits the rest of that text: those later words might be part of the credential, not an explanation. The same rule applies to copied -history errors, saved run messages and notebook cell output. Scheme-less text is cut only in +history errors, saved run messages and notebook cell output. Scheme-less text is scanned as an address in copied history errors and the project log's saved parameters and errors (Copy as code keeps opaque text parameters and says so, as below): a secret in the query or fragment of one of its words (`data.tif?token=abc def`, `a.tif?x=1;token=abc def`) goes with the rest of the text, saying `[query and the rest omitted]`. A word alone (`data.tif?pwd=hunter2`, `data.tif#pwd=hunter2`) has only the secret's value emptied, with no marker. A `key`, `sig` or `sas` counts only when its value reads as a key, -so `a.tif?key=abc def` stays. Notices and notebook output cut only `scheme://` addresses and -keep scheme-less text as written. For an unknown +so `a.tif?key=abc def` stays. Notices and failed notebook command output also apply the +credential-field rules to scheme-less text: `a.tif?token=abc def` loses the value and the rest +of the text. Successful notebook command output removes recorded login bindings, recognized +addresses and known login secrets, but can keep opaque scheme-less text such as that example. +For an unknown `scheme://` URL, a query with no secret name (`?custom=hidden unescaped hidden`) is omitted only to its first space: the redactor can't tell a value's later words from prose, so they are kept. Ambiguous slash-containing userinfo is omitted rather than guessed to be a host, port and path. diff --git a/docs/user-guide/analysis.md b/docs/user-guide/analysis.md index 1db2b4b912..f23385b450 100644 --- a/docs/user-guide/analysis.md +++ b/docs/user-guide/analysis.md @@ -686,8 +686,9 @@ layers**: the prediction, named by `output`, and `_variance`. the count of output polygon features whose total area is less than one millionth (`1e-6`) of their input feature's area, measured in the overlay plane. These may be slivers along the clip boundary; they are kept unchanged. Check these features before using the result as a map series -index, where a tiny sliver can make an apparently empty page. Ordinary clipped pieces give no -sliver notice, and empty intersections are still left out. +index, where a tiny sliver can make an apparently empty page. A genuine small piece of a very +large input can also trigger this notice. Pieces at or above the ratio give no sliver notice, +and empty intersections are still left out. **Erase** (`erase`) removes from `input` features (points, lines or polygons) their parts inside the `erase` polygons. An erase polygon's edge counts as inside it, as for Clip, so a point on the @@ -948,9 +949,9 @@ digits, so `06/23/2021` converts but `6/23/2021` doesn't). A date such as `2026-03-05` becomes `2026-03`, keeping different years separate. 2. Run **Summary Statistics** on the permits, with **Case Fields** = `issue_month` and a **COUNT** statistic on `issue_date` (use **Add statistic**). -3. Read `FREQUENCY` as the number of permits per month. Records without a date form a separate - group with an empty key. The output - contains months present in the data; it does not fill in missing months with zero. +3. Read `FREQUENCY` as the number of permits per month. A null date and an empty-text date form + separate groups with null and empty-text keys; if both forms occur, there are two groups. The + output contains months present in the data; it does not fill in missing months with zero. **Minimum Bounding Geometry** (`convexHull`) encloses each feature in its convex hull (Group Option `None`), or all of them in one (`All`, with `count`, the features it enclosed). It passes @@ -1741,9 +1742,12 @@ The Imagery tab's groups open their tools in the Workflows pane. The tab belongs (each channel's most frequent value among the pixels that show). Class names are kept with `nearest` and `mode`; other methods remove them with a notice naming the affected bands. Palette images use `nearest` for sampling and `mode` for aggregation, keeping their classes. - - Cells off the raster are its no-data value. A raster without one keeps its bands as they are, - with NaN there in a floating-point type and 0 in a whole-number one, as GDAL leaves them; a - colour image gains an alpha band instead, so they are see-through. + - Cells off the raster use 0 in every band when the image has alpha or gains it, so they are + see-through. Otherwise an explicit no-data value takes precedence. Without either, every + band uses NaN when any band has class names, the type is floating-point, or RGB data has + more than three bands (for example RGBI, whose fourth band is data). Other unlabelled + whole-number data uses 0. A three-band RGB image without alpha or no-data gains an alpha + band for its outside cells. - Datums are shifted as everywhere in RustGIS, through each system's transformation to WGS 84 (a Helmert step for SAD69 or Córrego Alegre, a few metres from the IBGE's grid shifts). - The result has to fit in memory (120 million values, cells times bands, or more on a computer diff --git a/docs/user-guide/data-formats.md b/docs/user-guide/data-formats.md index 9ed5b41884..4f36dcf1d8 100644 --- a/docs/user-guide/data-formats.md +++ b/docs/user-guide/data-formats.md @@ -20,7 +20,7 @@ export names its feature. | --- | --- | --- | --- | | GeoJSON | `.geojson`, `.json` | Home › Add › Add Data, or drop the file | Features with typed attributes; a single-member `GeometryCollection` is unwrapped (including nested single-member collections). Empty or multi-member collections are unsupported: the whole feature, including its attributes, is skipped and counted in an unreadable-geometry notice. A FeatureCollection's legacy `crs` member (`{"type": "name", "properties": {"name": "urn:ogc:def:crs:EPSG::28992"}}`) is honoured for EPSG codes (`EPSG:28992`, `urn:ogc:def:crs:EPSG::28992`, `urn:ogc:def:crs:EPSG:6.6:28992`) and for `CRS84`: a projected or other non-WGS 84 system converts the coordinates to longitude/latitude and the layer keeps it as its coordinate system (Layer properties, export in it, Define Projection); `CRS84` and `EPSG:4326` are longitude/latitude already, so the coordinates stay as they are and the layer has no other coordinate system. Coordinates are x/y (easting/northing, or longitude/latitude) whatever the EPSG axis order. A `crs` that isn't understood (a `link`, WKT or PROJ text, any other name, an unknown code) is named in a notice and the coordinates are read as longitude/latitude. A feature whose coordinates can't be converted, or convert to beyond ±180.5° longitude or ±90.5° latitude (the declared system doesn't fit the numbers), is left out and counted in a notice, and Define Projection then asks for the file to be added again. A feature read as longitude/latitude (no `crs`, `CRS84` or `EPSG:4326`, or a `crs` that isn't understood) with a position whose longitude is within ±180.5° but whose latitude is past ±90.5° is kept, and one notice counts them: longitude and latitude may be swapped. A `crs` on a feature, a geometry or a lone Feature is ignored. GeoJSON is always written in WGS 84 without a `crs`. Desktop FeatureCollections larger than 32 MiB use an in-place reader (which honours `crs` too, reading the file once more to check the features convert); layers with more than 100,000 features stream a view at a time. Other GeoJSON files load whole, up to 256 MiB. | | Delimited text | `.csv`, `.tsv`, `.txt`, recognised `.dat`; UTF-8 `.csv.gz`, `.tsv.gz`, `.txt.gz`, `.dat.gz` | Add Data, or Home › Add › Add Data › Delimited Text… to choose the columns from the file's header | Comma, semicolon, tab or pipe separated (found from the header, or `"delimiter"`); `info.fileColumns` lists the columns and the ones found. Points from x/y columns (named like longitude/latitude, GTFS `stop_lon`/`stop_lat` or easting/northing, or chosen with `"x"` and `"y"`), or any geometry from a WKT/EWKT column (named like `wkt` or `geometry`, or `"wkt"`). When an inferred pair exceeds degree ranges, a recognised longitude/latitude pair is preferred only if its in-range, non-0,0 points cover at least as many rows as the first pair can place, unless a projected `"crs"` is given (see [Delimited Text](data.md) for aliases and sampling). Coordinates in another system are converted from `"crs"` (`EPSG:2263`, or WKT) or an EWKT `SRID=`; rows still read outside longitude/latitude are counted in a notice. Points placed at exactly 0,0 are kept and counted in a notice, as they may represent missing coordinates. Uncompressed text is UTF-8, or UTF-16 with its byte-order mark; other uncompressed files are read as Windows-1252 (Latin-1, as many public sources still publish CSV), with a notice saying so. Gzip input is UTF-8 only, streamed within 256 MiB of expanded text; decompress other encodings and add as `.txt`. Retained CSV rows and cells have a separate estimated memory budget (256 MiB in the browser; a quarter of physical memory between 256 MiB and 4 GiB on desktop). A file without coordinates opens as a standalone table. | -| Shapefile | `.shp` with `.dbf` (and `.shx`), or zipped (`.zip`) | Add Data or drop the file; in the web app, pick or drop the `.shp` and `.dbf` together, with the `.shx` and `.prj` if there are (they open as one layer), or the zip | A zip may hold several Shapefiles, in folders, and other vector files (GeoJSON, CSV, GeoParquet, FlatGeobuf); each becomes a layer, at most 32 for the whole archive (Shapefiles first, so other files are left out when Shapefiles fill it). A `.dbf` table without a `.shp` of the same name in its folder (such as a flood map's `L_` tables) isn't read, and a notice names it. Delimited `.txt` members (including GTFS tables) are read; unrecognised text members and single-column `README.txt` documentation are skipped with a notice and macOS `__MACOSX` entries are passed over, and a member that can't be read is skipped and named in a notice. Converted to longitude/latitude from the coordinate system in its `.prj` (EPSG, Esri or OGC WKT, or WKT2 as PROJ and QGIS write it; Web Mercator, UTM, State Plane and other common projections); without a `.prj` it must be in longitude/latitude. On the desktop, one with more than 100,000 shapes (counted from its `.shx`) is read in place, a view at a time. Numeric (`N`) and floating point (`F`) fields are read at full double precision. Date (`D`) fields are read as dates; a blank or zero date is empty, and one that isn't a date (`20260230`) is empty and counted in a notice. | +| Shapefile | `.shp` with `.dbf` (and `.shx`), or zipped (`.zip`) | Add Data or drop the file; in the web app, pick or drop the `.shp` and `.dbf` together, with the `.shx` and `.prj` if there are (they open as one layer), or the zip | A zip may hold several Shapefiles, in folders, and other vector files (GeoJSON, CSV, GeoParquet, FlatGeobuf); each becomes a layer, at most 32 for the whole archive (Shapefiles first, so other files are left out when Shapefiles fill it). A `.dbf` table without a `.shp` of the same name in its folder (such as a flood map's `L_` tables) isn't read, and a notice names it. Delimited `.txt` members (including GTFS tables) are read; unrecognised text members and single-column `README.txt` documentation are skipped with a notice and macOS `__MACOSX` entries are passed over, and a member that can't be read is skipped and named in a notice. Converted to longitude/latitude from the coordinate system in its `.prj` (EPSG, Esri or OGC WKT, or WKT2 as PROJ and QGIS write it; Web Mercator, UTM, State Plane and other common projections); without a `.prj` it must be in longitude/latitude. On the desktop, one with more than 100,000 shapes (counted from its `.shx`) is read in place, a view at a time. Numeric (`N`) and floating point (`F`) fields are read at full double precision; on whole-file reads, `nan` or `inf` becomes empty and is counted in a notice. Date (`D`) fields are read as dates; a blank or zero date is empty, and one that isn't a date (`20260230`) is empty and counted in a notice. | | KML / KMZ | `.kml`, `.kmz` | Add Data, drop the file, or upload it in the web app | A layer for each kind of geometry (points, lines, polygons), with each placemark's name, description, folder path and extended data (a key named like the `name`, `description`, `folder` or `style` column, ignoring case, gets a column of its own, such as `NAME_1`, unless it only repeats that column). Styles are kept: the layer is drawn by unique values of its `style` field, with each KML style's line and fill colours, widths and icon colours (up to 64 styles). Multi-geometries and `gx:Track`s are read. Ground overlays (pictures on a `LatLonBox`, turned by its `rotation`) are raster layers kept in the project: the picture inside a KMZ, or a file beside a KML (not a web address, nor outside its folder); a `gx:LatLonQuad` isn't drawn. | | GPX | `.gpx` | Add Data, drop the file, or upload it in the web app | Waypoints, routes, tracks (one line per track, its segments as parts) and track points (with elevation and time) as separate layers. | | Spreadsheets | `.xlsx`, `.xlsm`, `.xlsb`, `.xls`, `.ods` | Add Data (or Delimited Text… to choose the columns), drop the file, or upload it in the web app | One sheet at a time, read like delimited text: the first non-empty row names the columns, and x/y or WKT columns (found, or given with `"x"`, `"y"`, `"wkt"` and `"crs"`) make the geometry; a sheet without them is a table. The first sheet is added and the others named in a notice; `"layer"` picks one, and `info.fileLayers` lists them. Dates read as ISO 8601. | @@ -33,7 +33,7 @@ export names its feature. | ArcGIS feature-service answers (protocol buffer) | `.pbf` holding an answer to an ArcGIS query with `f=pbf`, or the query's address | Add Data, drop the file, upload, or From URL… with the query's address (`…/FeatureServer/0/query?…&f=pbf`) | One page of the layer: the fields and values the same query gives as GeoJSON, with the fields' aliases as their display names (dates as milliseconds since 1970, date-only and time values as text), points, lines or polygons converted to longitude/latitude from the answer's coordinate system. Z and M values, geometry, blob and raster fields are left out, with a notice. A page that stopped at the service's record limit keeps its features, with a notice that the rest aren't in it. A query the service refuses (an empty answer), a count, ids or an extent, curves, multipatches and envelopes give an error. Told from OpenStreetMap data by its content. | | TopoJSON | `.topojson`, or `.json` | Add Data, drop the file, or upload | Shared arcs (quantized or not) are rebuilt into geometries. A `.topojson` adds a layer per object, up to 64; a topology named `.json` adds its first object and names the others in a notice. Use `.topojson` to add them all. | | CityJSON | `.city.json`, or `.json` | Add Data, drop the file, upload, or From URL… | CityJSON 1.1 and 2.0, read in 2D: a layer per City Object type and geometry family (a name two layers would share, such as two long types cut to their first 64 characters, or two that differ only in ASCII case, gets ` (2)`, ` (3)`… on the later ones: the first number no other layer's name uses). Surfaces and solids become footprints (LoD 0, else the `GroundSurface` surfaces, else the union of the faces of the highest LoD, each tried when the one before gives no area, else the outline of the points; lines and points come from their highest level of detail that draws; a level passed over because its X or Y isn't a finite number, or is too large, is named in a notice; a geometry of more than 200,000 faces is the outline of its points (of its ground's, for a footprint from the ground), with a notice, unless its footprint is its ground and all of its ground faces are within the cap), with `z_min`, `z_max` and `height` fields, from every level of detail and the children's heights included, so Style › Extrusion stands them up, and `base_height`, how far an object's lowest point is above the lowest point of the topmost object it is part of (0 for a building; for a balcony part, its lowest point's height above the building's; a building's lowest point is its basement's when it has one, and a group's members aren't parts of the group). Also `id`, `type`, `parent`, `lod` and the object's attributes; one named as one of these fields, or as another attribute of its layer but for case, takes a number (`height_1`), the same in every row, and a notice lists them. A `.city.json`, or a model from an address, adds every layer, up to 64; a file named `.json` adds its first layer with something to draw and names the others in a notice. A file that names no coordinate system and isn't longitude/latitude is refused: for now the way to add it is the `"crs"` parameter of `layer.addData`, from the command line, Python or the AI Assistant. Geometry instances are their template placed by their matrix at their reference point, with its heights, and rings the matrix flattens are left out and said for each instance; one whose template is missing, can't be read or draws nothing, or whose matrix is missing or isn't 16 numbers ending in `0 0 0 1`, is drawn as a point at its reference point, with no height, as are those past what placing may cost (as many points, all instances together, as the file could write out), and a notice says how many. `TINRelief` objects aren't drawn; a notice says so, as it does for addresses. Appearance (materials and textures) isn't read, and a notice says how much is left out. Heights are kept in the file's unit: in a coordinate system that doesn't measure in metres, a notice says so, as Style › Extrusion reads metres. A compound URN (`urn:ogc:def:crs,crs:EPSG::25832,crs:EPSG::5783`) is read by its horizontal part, and its vertical part gives the heights' unit, said as in CityGML when it isn't a height in metres. In a zip, a `"crs"` applies only to a model that names no coordinate system RustGIS knows. In the browser a footprint that needs a union is the outline of the object's points; a notice says so. An extrusion raises each part when it is given a base, such as `base_height` as its base field, or a constant (see [Layer effects](cartography.md#layer-effects)); with no base, parts stand on the ground. | -| CityGML | `.gml`, or `.xml` | Add Data, drop the file, upload, or From URL… | CityGML 1.0, 2.0 and 3.0, read in 2D as CityJSON is (the same layers, fields, footprints, heights and notices). 3.0 is read by the names it keeps and by its own where they change (thematic surfaces in `boundary`, openings in `fillingSurface`, generic attributes named by `gen:name`, a group's member inside its `Role`), read only in 3.0's namespaces, so an extension of a 1.0 or 2.0 model that uses one of those names keeps its own meaning; its spaces (a `BuildingRoom`, a `Storey`) are objects like any other. The objects are the features of the model's members and those nested in them with a `gml:id` or a geometry (a `BuildingPart`, a `Room`), each with its parent's id in `parent`; thematic surfaces and their openings are their object's faces, and a `GroundSurface` its ground. A geometry's level of detail is the one its property names (`lod2Solid` is 2); one in another property (a group's) has none. Converted from the `srsName`, by its horizontal part for a compound system (the URN and URL forms of geographic systems are latitude first, CRS84 longitude first; heights are kept as they come, and a vertical part that isn't a height in metres, such as one in feet, a depth, or one RustGIS doesn't know, is said; when a vertical part is named, its unit is the heights', not the horizontal system's), or from the coordinate system given (`crs`), which takes the place of the one the file names; a file naming different systems (on its geometries or envelopes) is refused unless a system is given, which reads them all and says how many named another, and one system with its axes in different orders is refused either way. Picked by name (`layer`), a layer keeps the notices the file as a whole gives; zipped, a model keeps the system it names, and the one given applies only when it names none RustGIS knows. Generic attributes are typed by their element, `measuredHeight`, the storey counts, years and any property with a unit in metres are numbers (a measure in another unit than the metre is text with its unit), other simple properties are text (the code list a code names, `codeSpace`, a value's other attributes, such as its language, `xml:lang`, and a generic attribute set's members with no name and its own code list aren't kept, and notices say how many, generic attributes included), a property given twice is a list and a structured one its JSON. Each member of the model is parsed on its own, with the objects nested in it (each object of a `featureMembers` on its own), so no tree of the whole model is built; the file's text and the rows read stay in memory. On the desktop, Add Data reads a file added by its path in the background, with its progress counted a member at a time and Cancel (in a zip, from text, or listed, it is read as the command runs). A member nested too deep or too large to parse is left out whole, with a notice. A geometry given by a link (`xlink:href`) to one written in the file, in the same member or another, as a solid's faces are to its walls' and roof's polygons, is read where the link stands, once its object's own geometries are read, a ground surface's links first: one its object holds already at that level of detail isn't read again (a ground face stays its ground, and a face read as a wall that a ground surface links to is read as the ground too), through links each is read at most once per object and level, so links in a cycle end, and what a member's links read is bounded by the member's size, and what links copy and read from other members by four times the file's, past which the links left are said; a solid whose exterior is a link that is followed and reads something keeps its cavities; a link to a `gml:id` that several elements have reads the one in its member, else the first in the file, and a notice says how many geometries repeat one; a geometry whose coordinates can't be read is said once, whichever object or link reads it first. An implicit geometry is its prototype (`relativeGMLGeometry`, `relativeGeometry` in 3.0, written in place or given by a link, in its member or another) placed by its matrix at its reference point, in the file's axis order (latitude first where the system is), with its heights: the prototype is read once and placed for each instance, and rings its matrix flattens are left out and said for each instance; one whose prototype is in another file (`libraryObject`), can't be read or holds nothing to place, or whose matrix isn't 16 numbers ending in `0 0 0 1`, is drawn as a point at its reference point, with no height, as are those past what placing may cost (as many points, all instances together, as the file could write out), and a notice says how many. A part, a group's member or a group's parent given by a link (`consistsOfBuildingPart`, `buildingPart`, `bridgePart` and `tunnelPart` in 3.0, `groupMember`, a `CityObjectGroup`'s `parent`) makes the object it names a child of the other, as if written in it, so its heights go to it, wherever it is in the file (an object written in a group's `parent` is the group's parent too, not its child); the link stays an attribute too, a link naming no other object of the file is said, and a member's role in its group isn't kept, and is said. Not read, and said in a notice: geometries given by a link to another file, nothing or a member that can't be parsed, to what isn't a geometry, or to one where nothing is drawn (an appearance's, a terrain intersection's), a prototype's geometry linked from outside its implicit geometry, and links inside a prototype; thematic surfaces and openings given only by a link; rings given by one (a hole so given is left out, its face kept), lines and rings with a point or curve given by one, and an oriented curve whose base curve is, terrain intersection curves, appearances, addresses, and the properties of thematic surfaces and openings (a name, generic attributes: their faces are kept as their object's); arcs and splines are read as straight lines through their points (a curve given backwards, `orientation="-"`, in its direction, and one held in a ring counted like a standalone one); an arc given by its centre (`ArcByCenterPoint`, `CircleByCenterPoint`) isn't read as the curve's points, its centre is never a vertex, and an arc that is its curve's only segment leaves the geometry with nothing to read; a `gml:Point` written with more than one position keeps the first one, and is said; a solid with no exterior shell is left out, as is a solid whose exterior has nothing read (a link that isn't followed, an empty shell, or a shell whose surfaces are all links that lead nowhere), its cavities with it (read alone they would be drawn as a footprint), and one whose cavity can't be read is left out whole; a file cut short, even inside a tag, keeps the objects before the cut; `TINRelief` objects aren't drawn; 3.0's relations between objects (`CityObjectRelation`) and the model's versions (`versionMember`, `versionTransitionMember`) aren't read. | +| CityGML | `.gml`, or `.xml` | Add Data, drop the file, upload, or From URL… | CityGML 1.0, 2.0 and 3.0, read in 2D as CityJSON is (the same layers, fields, footprints, heights and notices). 3.0 is read by the names it keeps and by its own where they change (thematic surfaces in `boundary`, openings in `fillingSurface`, generic attributes named by `gen:name`, a group's member inside its `Role`), read only in 3.0's namespaces, so an extension of a 1.0 or 2.0 model that uses one of those names keeps its own meaning; its spaces (a `BuildingRoom`, a `Storey`) are objects like any other. The objects are the features of the model's members and those nested in them with a `gml:id` or a geometry (a `BuildingPart`, a `Room`), each with its parent's id in `parent`; thematic surfaces and their openings are their object's faces, and a `GroundSurface` its ground. A geometry's level of detail is the one its property names (`lod2Solid` is 2); one in another property (a group's) has none. Converted from the `srsName`, by its horizontal part for a compound system (the URN and URL forms of geographic systems are latitude first, CRS84 longitude first; heights are kept as they come, and a vertical part that isn't a height in metres, such as one in feet, a depth, or one RustGIS doesn't know, is said; when a vertical part is named, its unit is the heights', not the horizontal system's), or from the coordinate system given (`crs`), which takes the place of the one the file names; a file naming different systems (on its geometries or envelopes) is refused unless a system is given, which reads them all and says how many named another, and one system with its axes in different orders is refused either way. Picked by name (`layer`), a layer keeps the notices the file as a whole gives; zipped, a model keeps the system it names, and the one given applies only when it names none RustGIS knows. Generic attributes are typed by their element, `measuredHeight`, the storey counts, years and any property with a unit in metres are numbers (a measure in another unit than the metre is text with its unit), other simple properties are text (the code list a code names, `codeSpace`, a value's other attributes, such as its language, `xml:lang`, and a generic attribute set's members with no name and its own code list aren't kept, and notices say how many, generic attributes included), a property given twice is a list and a structured one its JSON. Each member of the model is parsed on its own, with the objects nested in it (each object of a `featureMembers` on its own), so no tree of the whole model is built; the file's text and the rows read stay in memory. On the desktop, Add Data reads a file added by its path in the background, with its progress counted a member at a time and Cancel (in a zip, from text, or listed, it is read as the command runs). A member nested too deep or too large to parse is left out whole, with a notice. A geometry given by a link (`xlink:href`) to one written in the file, in the same member or another, as a solid's faces are to its walls' and roof's polygons, is read where the link stands, once its object's own geometries are read, a ground surface's links first: one its object holds already at that level of detail isn't read again (a ground face stays its ground, and a face read as a wall that a ground surface links to is read as the ground too), through links each is read at most once per object and level, so links in a cycle end, and what a member's links read is bounded by the member's size, and what links copy and read from other members by four times the file's, past which the links left are said; a solid whose exterior is a link that is followed and reads something keeps its cavities; a link to a `gml:id` that several elements have reads the one in its member, else the first in the file, and a notice says how many geometries repeat one; a geometry whose coordinates can't be read is said once, whichever object or link reads it first. An implicit geometry is its prototype (`relativeGMLGeometry`, `relativeGeometry` in 3.0, written in place or given by a link, in its member or another) placed by its matrix at its reference point, in the file's axis order (latitude first where the system is), with its heights: the prototype is read once and placed for each instance, and rings its matrix flattens are left out and said for each instance; one whose prototype is in another file (`libraryObject`), can't be read or holds nothing to place, or whose matrix isn't 16 numbers ending in `0 0 0 1`, is drawn as a point at its reference point, with no height, as are those past what placing may cost (as many points, all instances together, as the file could write out), and a notice says how many. A part, a group's member or a group's parent given by a link (`consistsOfBuildingPart`, `buildingPart`, `bridgePart` and `tunnelPart` in 3.0, `groupMember`, a `CityObjectGroup`'s `parent`) makes the object it names a child of the other, as if written in it, so its heights go to it, wherever it is in the file (an object written in a group's `parent` is the group's parent too, not its child); the link stays an attribute too, a link naming no other object of the file is said, and a member's role in its group isn't kept, and is said. Not read, and said in a notice: geometries given by a link to another file, nothing or a member that can't be parsed, to what isn't a geometry, or to one where nothing is drawn (an appearance's, a terrain intersection's), a prototype's geometry linked from outside its implicit geometry, and links inside a prototype; thematic surfaces and openings given only by a link; rings given by one (a hole so given is left out, its face kept), lines and rings with a point or curve given by one, and an oriented curve whose base curve is, terrain intersection curves, appearances, addresses, and the properties of thematic surfaces and openings (a name, generic attributes: their faces are kept as their object's); arcs and splines are read as straight lines through their points (a curve given backwards, `orientation="-"`, in its direction, and one held in a ring counted like a standalone one); an arc given by its centre (`ArcByCenterPoint`, `CircleByCenterPoint`) isn't read as the curve's points, its centre is never a vertex, and an arc that is its curve's only segment leaves the geometry with nothing to read; a `gml:Point` written with more than one position keeps the first one, and is said; a solid with no exterior shell is left out, as is a solid whose exterior has nothing read (a link that isn't followed, an empty shell, or a shell whose surfaces are all links that lead nowhere), its cavities with it (read alone they would be drawn as a footprint), and one whose cavity can't be read is left out whole; a file cut short, even inside a tag, keeps completed members and completed direct children of the truncated member; a part nested inside an unfinished object is lost with that object; `TINRelief` objects aren't drawn; 3.0's relations between objects (`CityObjectRelation`) and the model's versions (`versionMember`, `versionTransitionMember`) aren't read. | | AutoCAD DXF | `.dxf` (ASCII) | Add Data › DXF…, drop the file, or upload | Points, lines and polygons layers from the drawing's entities: POINT, LINE, LWPOLYLINE and POLYLINE (bulges drawn as arcs; closed ones stay lines; polyface meshes become polygons), CIRCLE, ARC, ELLIPSE, SPLINE, HELIX, LEADER, MLINE (its centre line), TEXT, MTEXT, ATTRIB and constant ATTDEF (points with their text, at the alignment point of justified text), HATCH boundaries (holes found by nesting), SOLID, TRACE and 3DFACE (polygons). Blocks (named in any case) are drawn where INSERTs place them (scaled, rotated, in arrays, nested up to 8 deep), and so are DIMENSIONs' blocks (a linear dimension without one draws its lines and measured text); `"blocks": false` leaves them out, with a notice of how many. Entities in an object's own coordinate system are turned into the drawing's. Fields: `layer`, `entity`, `color` (ACI, BYLAYER and BYBLOCK resolved), `linetype`, `text`, `height`, `rotation`, `block`, `elevation` and `paperspace`. The drawing's layouts (paper space: title blocks, viewport frames) are left out, with a notice, unless `"paperSpace": true`. A drawing never says its coordinate system: give `"crs"` (such as `EPSG:2263`) — the import dialog offers the map's — or its coordinates are kept as they are, with a notice. `"layer"` picks one of the drawing's own layers, and `info.fileLayers` lists them. Text is read as UTF-8, and otherwise in the drawing's `$DWGCODEPAGE` (GBK, Big5, Shift_JIS, Windows-1252…), so Chinese layer names and labels come through; an unknown or missing code page is read as Latin-1, with a notice. Up to 2 million features (500,000 in the web app). Not read: binary DXF, MULTILEADER, WIPEOUT, IMAGE and ACIS solids (3DSOLID, REGION), named in a notice. | | Geotagged photos | `.jpg`, `.jpeg` | Add Data (several at once in the web app), drop them, upload, or `layer.addPhotos` with `"paths"` or a `"folder"` | JPEGs with a GPS position in their EXIF become points in the **Photos** layer (photos added one at a time join it): each with its `name`, `file`, `time` (when it was taken), `altitude`, `direction` (where the camera faced), `camera` and a `thumbnail` of at most 160 pixels, kept with the feature so the project needs no other file. Identify shows the thumbnail. Photos without a position are named in a notice and left out; a layer holds up to 2,000 photos. A JPEG is always read as a photo unless `layer.addData` or `layer.addDataBytes` is given `"as": "scan"` (a phone photo of a paper map, say: see [Scans with no coordinates](#rasters)). | | LiDAR point clouds | `.las`, `.laz` (COPC too) | Add Data, drop the file, upload, or From URL… | LAS 1.0–1.4 (point formats 0–10) and LAZ, on desktop and web: points with `z`, `intensity`, `classification`, `return` and colour (`red`, `green`, `blue`), coloured by ASPRS class or elevation. An evenly spread sample is read once, at most 2 million points (500,000 in the web app); views use that sample, not the COPC spatial index. LAZ decompression reads at most the first 400 million points (40 million on the web), with a notice when capped. Converted from the file's WKT or GeoTIFF-key record; without one, coordinates are kept with a notice. A local project keeps the path; uploads keep the file bytes. Remote files download whole. | @@ -90,7 +90,7 @@ place, is left out, and a notice names it. The parts are looked up beside the ad (S3's `X-Amz-*`) signs one file, so its Shapefile's parts can't be fetched with it: add that Shapefile zipped. Downloads are limited to 512 MiB on desktop and 256 MiB on the web; individual readers may have -lower limits (the whole-file vector readers use 256 MiB). In the web app the server must allow +lower limits (the whole-file vector readers, including CityGML, use 256 MiB). In the web app the server must allow requests from web pages (CORS). On the desktop, with [GDAL](https://gdal.org) installed, a JPEG 2000 file (`.jp2`, `.j2k`: Sentinel-2's imagery on S3) is downloaded too, converted to a GeoTIFF that the project keeps, and added. Desktop byte uploads diff --git a/docs/user-guide/data.md b/docs/user-guide/data.md index bb16e96829..e0a86677bd 100644 --- a/docs/user-guide/data.md +++ b/docs/user-guide/data.md @@ -54,10 +54,17 @@ first and its points last, so its points are drawn on top, and a KML's ground ov them. A name given goes to the first map layer added, the bottom one (a standalone table only gets it when the file has no map layer). A GeoPackage adds its tables in name order, and a file read through GDAL keeps its layers in the file's order and orders only the types within each layer. -GeoPackage attribute-only tables are not supported. Adding a whole file reports their count in -one notice. Tile and other unsupported GeoPackage contents are also counted by type. The omission counts cover catalog entries without a registered geometry column; unknown content types are combined into one counted "other" group. Only contents registered in `gpkg_contents` are considered; arbitrary unregistered SQLite tables are ignored. Desktop uploads add their first feature table, with the existing note naming how many feature tables the file contains. Browser uploads with virtual generated `table_name` or `data_type` catalog columns cannot be evaluated and are refused with an explanation; materialize these columns or select a feature table explicitly. A file -containing only attribute tables is refused with the count. Selecting a -feature table explicitly adds only that table. +GeoPackage attribute-only tables are not supported. Adding a whole file reports their count in one +notice. Tile and other unsupported GeoPackage contents are also counted by type. The omission counts +cover catalog entries without a registered geometry column; unknown content types are combined into +one counted "other" group. Only contents registered in `gpkg_contents` are considered; arbitrary +unregistered SQLite tables are ignored. Desktop uploads add their first feature table, with the +existing note naming how many feature tables the file contains. Browser uploads with virtual +generated `table_name` or `data_type` catalog columns cannot be evaluated and are refused with an +explanation; materialize these columns or select a feature table explicitly. A file containing only +attribute or tile tables is refused with the count. Selecting a feature table explicitly adds only +that table. + On the desktop, **Home › Add › Add Data › Add Folder…** connects a folder to the Browser; the web app does not offer folder connections. On the desktop, a file Add Data reads whole (a GeoTIFF or another raster, one GDAL converts, a @@ -100,10 +107,11 @@ a specific longitude/latitude pair is preferred over generic X/Y when it gives i non-0,0 positions on at least as many rows as the first pair gives finite positions. That pair is used for the whole file. The published aliases `Longitude Decimal Degrees`/`Latitude Decimal Degrees` and -`Longitutde`/`Lattitude` are recognised too. The dialog checks its first 1,000 rows to pick -defaults, and recomputes unchanged defaults when you leave the coordinate-system box or press -Enter; columns you picked stay picked. With a projected coordinate system, defaults prefer -X/Y or easting/northing columns when the file also has longitude/latitude columns. +`Longitutde`/`Lattitude` are recognised too. The dialog checks the first 1,000 rows of delimited +text, or every row of a spreadsheet, to pick defaults. It recomputes unchanged defaults when you +leave the coordinate-system box or press Enter; columns you picked stay picked. With a projected +coordinate system, defaults prefer X/Y or easting/northing columns when the file also has +longitude/latitude columns. Add Data checks all rows. Coordinates still outside degree ranges are counted in a notice. Points placed at exactly 0,0 are also counted in a notice: these often mean missing coordinates, but the points are kept. A CSV without coordinate columns, or a layer without geometry, opens as a @@ -118,11 +126,12 @@ the project, from two fields of coordinates and their coordinate system. In a te Excel `.xls` records, formula operands and sheet bounds are checked before opening, including sheets that were not selected. The workbook shares a memory budget across its sheets and retained -metadata. Value and formula grids are counted separately, with conservative estimates for -formula strings, defined names and sheet state. Formula-heavy workbooks can reach this budget -before their value-grid limit. A file with damaged records or excessive allocations is refused with the reason. Normal -cached formula values are read. A formula's direct cell reference to row 65,536 is currently -refused because the XLS reader cannot handle that reference safely. +metadata. Value and formula grids are counted separately, with conservative allocation estimates +for formula strings, defined names and sheet state. Formula-heavy workbooks can reach this budget +before their value-grid limit. A file with damaged records or excessive allocations is refused +with the reason. Normal cached formula values are read. A formula's direct cell reference to +row 65,536 is currently refused because the XLS reader cannot handle that reference safely. + **Home › Add › Add Data › Graphics Layer** adds a layer for [drawn graphics](cartography.md#graphics), and **Data › Layer Templates** adds empty layers with ready-made schemas to draw into. @@ -658,7 +667,8 @@ plus `geocoded`, what the geocoder called the place, and a text field `match_sta Rows that weren't found stay in its table without a point, and a notice counts them; filter `match_status = 'unmatched'` to review them. A field of the same name in the input is kept, and -the new one gets an underscore appended (`match_status_`; filter that one then). `matched` means the geocoder returned a point, not that +the generated field gets underscores appended until its name is unique (`match_status_`, +`match_status__`, and so on); filter the generated field. `matched` means the geocoder returned a point, not that it is the exact address: it may be placed along the street or at a broader place. The geocoders don't give a match score, so there is none in the output. diff --git a/docs/user-guide/editing.md b/docs/user-guide/editing.md index 1f510b90ec..f60d05918c 100644 --- a/docs/user-guide/editing.md +++ b/docs/user-guide/editing.md @@ -370,7 +370,7 @@ opens. A picture from any other server isn't: the pop-up shows **Show Picture** name, and fetches nothing until you click it, because the request tells that server that someone opened the data (see [Privacy](../privacy.md)). A spinner shows while a picture is fetched. PNG, JPEG, WebP and GIF pictures of up to 4 MB and 2048 × 2048 pixels are shown (on every platform; any other -format, such as BMP or TIFF, is "not supported"); the last 16 addresses' pictures are kept for the +format, such as BMP or TIFF, shows "Picture unavailable: … pictures aren't supported"); the last 16 addresses' pictures are kept for the session, so opening the pop-up again shows the picture without asking again; they are forgotten with the web layers' cached tiles (signing out of ArcGIS, a login that ends, **Reload Web Layers**). When one can't be shown, the pop-up says why ("Picture unavailable: …": the server's answer, a network diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 6655be8ead..5e49eaa30c 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -118,7 +118,7 @@ end, by double-clicking a tab, or with ++ctrl+f1++ (++cmd+f1++ on macOS, where F key). Click a tab to peek at its commands while collapsed. When the window is narrower than the ribbon, arrows at its ends scroll it. -### Diagnostics +### Messages Open **View › Windows › Messages**, click the status-bar notice or **Messages** button, or search for **Messages** in Command Search (`ui.pane.diagnosticLog`). This troubleshooting @@ -273,7 +273,7 @@ in Command Search. Reload rereads the same local source, table or raster subdata adding other tables from its container. It retains the layer's name, identity, drawing order, group, visibility, opacity, styling, labels, filters and map camera. Local PMTiles and MBTiles tile archives have their own **Reload Data** on their Layers row; see -[Maps › Reloading a local tile archive](maps.md). +[Maps › Reloading a local tile archive](maps.md#reloading-a-local-tile-archive-desktop). Supported targets are linked vectors, rasters (including time-following Zarr variables), GDAL/DuckDB file-backed layers and editable local GeoPackage tables. GDAL-linked File diff --git a/docs/user-guide/maps.md b/docs/user-guide/maps.md index d5fa6bd099..a29685b8e3 100644 --- a/docs/user-guide/maps.md +++ b/docs/user-guide/maps.md @@ -462,7 +462,9 @@ selectable or editable; a tile not yet loaded may need another click once it arr **Reload** in the Basemap gallery (`map.reloadWebLayers`) clears cached tiles and styles and fetches them again. -**Reloading a local tile archive (desktop).** After changing a PMTiles or MBTiles file on disk, +### Reloading a local tile archive (desktop) + +After changing a PMTiles or MBTiles file on disk, right-click its row in Layers and choose **Reload Data** (**Reload Tile Archive** in Command Search). CLI, Python, MCP and the Assistant use `map.reloadTiles={"index":0}`; `"name"` or catalog `"id"` can select the archive instead. @@ -510,6 +512,10 @@ every tile payload. It allows at most **64 MiB stored and 64 MiB decoded index b index entries**, **4,096 unique directories** (including the root), and **four leaf levels**; each stored directory read is limited to **16 MiB**. An invalid index or an exceeded validation budget is an explicit error without eviction, even if an unusually large index is otherwise valid. +Validation also refuses a stored tile range over **16 MiB**, or one outside the header's declared +tile-data section (including a zero-length section with nonempty tile entries). Ordinary loading +can still draw other tiles from such an archive; reload and repair require the replacement's +entire index to pass these checks before changing the source or dropping cached data. Neither action is available for local archives in the web app, or for remote tile services. On Windows, an open MBTiles file may not be replaceable while RustGIS holds it open: reload diff --git a/docs/user-guide/sharing.md b/docs/user-guide/sharing.md index c17adcbf59..99fa4d5c45 100644 --- a/docs/user-guide/sharing.md +++ b/docs/user-guide/sharing.md @@ -52,7 +52,7 @@ vectors as a layout's PDF is (see Layouts), in the system viewer on desktop; **C Clipboard** copies a map image. **Export To 3D** writes a layer as a Wavefront OBJ or binary STL model with polygons extruded by a field (`share.export3d` writes STL when `path` ends in `.stl`). STL holds closed solids only: lines and zero-height features are left out, zero-area triangles are -skipped, and a cap that can't be triangulated is fanned over its outer ring, filling any holes; one +skipped, and a cap that can't be triangulated is fanned over its outer ring, filling any holes; a cap that the triangulator can only finish as a fan (a hole touching the outer ring along an edge, for example) is fanned over what is left of it. Either fan may spill outside the polygon; the notices say how many. OBJ