Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
0b5b2c8
feat(engine): batch geoprocessing runner (gp.batch)
quanguet0409 Nov 5, 2026
c513219
fix(engine): address bot review comments and resolve CI test failures…
quanguet0409 Nov 5, 2026
1a9ebe6
fix(engine): address macro progress, security gate, and input notices…
quanguet0409 Nov 5, 2026
0df3ea3
fix(engine): track actual result layer names and preserve updated lay…
quanguet0409 Nov 5, 2026
afa5af7
fix(engine): address in-flight cancellation, notice aggregation, and …
quanguet0409 Nov 5, 2026
fa1cb80
fix(engine): reject file-writing tools, aggregate tool messages, and …
quanguet0409 Nov 5, 2026
d1dc850
Merge branch 'main' into feature/gp-batch
quanguet0409 Nov 5, 2026
658623a
fix(engine): resolve output collisions, shapefile sidecars, quiet too…
quanguet0409 Nov 5, 2026
979c28e
fix(engine): ensure wasm32 compatibility without unused warnings in g…
quanguet0409 Nov 5, 2026
fadd21d
fix(engine): skip symlinks in folder scan, gate geolibre batch tools,…
quanguet0409 Nov 5, 2026
c9f1b04
fix(engine): report table outputs in batch, distinguish subfolders, a…
quanguet0409 Nov 5, 2026
bb00985
fix(engine): record item error before breaking on batch cancellation
quanguet0409 Nov 5, 2026
a9019fe
fix(engine): exclude cancelled item from failures and check batch par…
quanguet0409 Nov 5, 2026
f6b5068
fix(engine): handle table collisions, complete sidecars, and consolid…
quanguet0409 Nov 5, 2026
e377f05
fix(engine): address table collision restoration, sidecars scan, touc…
quanguet0409 Nov 5, 2026
caf8f10
fix(engine): add batch table output without replacing project tables …
quanguet0409 Nov 5, 2026
f24d5a6
fix(engine): report table renames via apply_inner and collect Output:…
quanguet0409 Nov 5, 2026
41e059e
fix(engine): do not record cancelled batch items as failures
quanguet0409 Nov 5, 2026
d56dd8d
fix(engine): record original target in gp_history, include selected l…
quanguet0409 Nov 5, 2026
b9d56b6
fix(engine): probe output collisions after temp layer creation and fo…
quanguet0409 Nov 5, 2026
da96ac4
fix(engine): emit notice for unused outputPattern and consolidate bat…
quanguet0409 Nov 5, 2026
9b6da4b
fix(engine): narrow batch file-access check to file inputs, warn on u…
quanguet0409 Nov 5, 2026
84e20fd
fix(engine): class all non-wildcard batch inputs as file access in re…
quanguet0409 Nov 5, 2026
6f60820
fix(engine): resolve GeoLibre tool ids in reaches, notify on overwrit…
quanguet0409 Nov 5, 2026
231babc
docs(analysis): clarify wildcard batch file-access conditions
quanguet0409 Nov 5, 2026
8284885
fix(engine): record file inputs as replayable batch commands, treat e…
quanguet0409 Nov 5, 2026
3d6e70d
fix(engine): record explicit param in file batch history, prevent tem…
quanguet0409 Nov 5, 2026
14ab4c7
fix(engine): define temp_layer_id as None on wasm32 for batch collisi…
quanguet0409 Nov 5, 2026
490dcd2
fix(engine): remove unused mut on wasm32 for temp_layer_id
quanguet0409 Nov 5, 2026
1ac17c9
fix(engine): treat null batch options as absent and emit no spurious …
quanguet0409 Nov 5, 2026
698f543
fix(engine): handle non-utf8 filenames in batch folder scan and refin…
quanguet0409 Nov 5, 2026
fc632f3
fix(engine): strip batch params.output from file access checks and ca…
quanguet0409 Nov 5, 2026
df2fb19
fix(engine): omit existing map layer selection from batch outputs and…
quanguet0409 Nov 5, 2026
49d1d08
fix(engine): aggregate distinct tool and reader messages by kind in b…
quanguet0409 Nov 5, 2026
acda179
fix(engine): emit specific error when batch wildcard matches no vecto…
quanguet0409 Nov 5, 2026
e2d627a
fix(engine): preserve raw input strings for layer lookup and notify o…
quanguet0409 Nov 5, 2026
c701c42
fix(engine): skip graphics layers in wildcard batch and unify batch f…
quanguet0409 Nov 5, 2026
d66d749
fix(engine): admit all native vector and tabular file formats in batc…
quanguet0409 Nov 5, 2026
b8df787
fix(engine): aggregate trimmed file path notices, separate pattern ke…
quanguet0409 Nov 5, 2026
4eb918a
fix(engine): gate trimmed_paths on non-wasm32 to fix unused_mut in wa…
quanguet0409 Nov 5, 2026
3a3a73e
fix(engine): trim file stems for output names, hoist is_file_input be…
quanguet0409 Nov 5, 2026
db2ac2f
fix(engine): handle blank folder parameter cleanly and document neste…
quanguet0409 Nov 5, 2026
7337e4d
fix(engine): notify on non-string folder, clarify all-fail batch sema…
quanguet0409 Nov 5, 2026
9909b1a
fix(engine): aggregate reader warnings, return error on zero successe…
quanguet0409 Nov 6, 2026
3bbbf93
fix(engine): skip folder fallback on empty wildcard and test raster-o…
quanguet0409 Nov 6, 2026
8159539
fix(engine): record failed history and preserve reader warnings on se…
quanguet0409 Nov 6, 2026
def7fcb
docs: update GEOLIBRE_PARITY.md for gp.batch
quanguet0409 Nov 6, 2026
a9d9939
fix(engine): enforce plain output name check for gp.batch in layer_na…
quanguet0409 Nov 6, 2026
5b463de
fix(engine): report requested total on cancel and warn on unrecognise…
quanguet0409 Nov 6, 2026
9abd002
fix(engine): preserve outputPattern on file read failure and document…
quanguet0409 Nov 6, 2026
48b5a46
fix(engine): avoid cloning parameters when unmodified and document na…
quanguet0409 Nov 6, 2026
f53531d
fix(engine): expose joinField aliases from analysis and notify on dro…
quanguet0409 Nov 6, 2026
bb9adde
fix(engine): preserve early return without history for gp.run missing…
quanguet0409 Nov 6, 2026
d9ba6c2
Merge origin/main into feature/gp-batch
quanguet0409 Nov 6, 2026
dc2dab5
fix(engine): drop unnecessary clone in reader warnings and clarify fo…
quanguet0409 Nov 6, 2026
b0665ed
fix(engine): accept tool parameter aliases for batch_param validation
quanguet0409 Nov 6, 2026
71be0d1
Merge branch 'main' into feature/gp-batch
quanguet0409 Nov 7, 2026
86c203a
Merge remote-tracking branch 'origin/main' into pr-1458
giswqs Nov 8, 2026
4ef7f01
docs(roadmap): batch runs are done once this merges
giswqs Nov 8, 2026
be5486f
Address review feedback
giswqs Nov 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions GEOLIBRE_PARITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,8 +196,8 @@ model-builder.md, segmentation.md, obia.md_
eCognition import. RustGIS has pixel classification.
- [ ] **Visual Model Builder canvas**: a node graph with validation and export to code. Partial:
ModelBuilder with a node canvas and models from history.
- [ ] **Re-run or copy history as code; batch runs over a folder**. Partial: history copied as
code; batch runs are missing (_Roadmap: solid ground_).
- [x] **Re-run or copy history as code; batch runs over a folder**: history copied as code, batch
runs over layers and folders (`gp.batch`).
- [ ] **Viewshed from public terrain tiles (no DEM needed)**. Partial: needs a loaded elevation
raster (_Roadmap: analysis_ for a DEM downloader).
- [x] **Conversion**: vector to PMTiles/GeoParquet, raster to COG: GeoParquet, COG and vector and
Expand Down
1 change: 0 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ Correct, robust and recoverable, with the foundations other themes stand on.
- Extend the OS keychain secret store to service connections (database passwords, S3 keys,
AI provider keys and the ChatGPT sign-in's tokens are already kept there).
- Vector tools, each writing a new layer: check and fix geometry _(open)_.
- Batch runs: one tool over many layers or a folder of files.

### Expressions and cartography
Next on top of the expression language:
Expand Down
26 changes: 26 additions & 0 deletions crates/analysis/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,32 @@ fn params_for(spec: &ToolSpec, params: &Value, built_in: bool) -> Result<Value,
Ok(Value::Object(p))
}

pub use tools::JOIN_FIELD_ALIASES;

/// Whether a parameter key is declared by `spec` or accepted as a valid parameter alias.
pub fn is_accepted_param(spec: &ToolSpec, key: &str) -> bool {
if spec.params.iter().any(|p| p.name == key) {
return true;
}
if spec.id == "joinField" { JOIN_FIELD_ALIASES.contains(&key) } else { false }
Comment thread
quanguet0409 marked this conversation as resolved.
Resolved
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
}

/// The names a tool reads one of its parameters under, `key` among them, or none when it has no
/// other names (Join Field takes its target as `target`, `input` or `target_layer`).
pub fn param_aliases(spec: &ToolSpec, key: &str) -> &'static [&'static str] {
const JOIN_FIELD: &[&[&str]] = &[
&["target", "input", "target_layer"],
&["joinTable", "join", "source", "join_table"],
&["targetField", "target_field", "key", "target_key"],
&["joinField", "join_field", "join_key"],
&["duplicates", "duplicateStrategy", "duplicate_strategy"],
];
if spec.id != "joinField" {
return &[];
}
JOIN_FIELD.iter().find(|group| group.contains(&key)).copied().unwrap_or(&[])
}

/// A built-in tool's on-off parameter as `true` or `false` (text saying so counts). Anything else is
/// an error, never a guess at what `"yes"` or `1` meant (#406). A blank value, or null, is the
/// default.
Expand Down
16 changes: 16 additions & 0 deletions crates/analysis/src/tools.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6168,6 +6168,22 @@ pub(crate) fn normalize_params(spec: &ToolSpec, p: &mut serde_json::Map<String,
}
}

/// Parameter aliases recognized by `normalize_join_field_params`.
pub static JOIN_FIELD_ALIASES: &[&str] = &[
Comment thread
quanguet0409 marked this conversation as resolved.
Resolved
"input",
"target_layer",
"join",
"source",
"join_table",
"target_field",
"key",
"target_key",
"join_field",
"join_key",
"duplicateStrategy",
"duplicate_strategy",
];

pub(crate) fn normalize_join_field_params(p: &mut serde_json::Map<String, Json>) {
let is_blank = |val: Option<&Json>| val.is_none_or(|v| v.is_null() || v.as_str().is_some_and(|s| s.trim().is_empty()));
let find_first =
Expand Down
3,556 changes: 3,525 additions & 31 deletions crates/engine/src/cmd/gp.rs

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion crates/engine/src/cmd/history.rs
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ fn script(s: &mut Session, _: &Value) -> CmdResult {
/// The commands other than `gp.run` a run is copied as. The history comes from the project file,
/// which may say anything: only these are written into code, so a crafted file can't pass any
/// other command off as a geoprocessing run (and an id is never text to escape).
const REPLAYED: &[&str] = &["gp.bufferPoint"];
const REPLAYED: &[&str] = &["gp.bufferPoint", "gp.batch"];
Comment thread
quanguet0409 marked this conversation as resolved.
Resolved

/// The command that runs `r` again, with its parameters: `gp.run` with the tool, or the command
/// that ran it (a point buffered where the map was clicked, `gp.bufferPoint`).
Expand Down
13 changes: 11 additions & 2 deletions crates/engine/src/policy/dispatch.rs
Original file line number Diff line number Diff line change
Expand Up @@ -693,10 +693,19 @@ impl Session {
}
/// Runs a pinned tool and records symbolic output identities for later frozen wires.
pub(crate) fn invoke_tool(&mut self, op: Authorized<ToolOp>, control: &Arc<rustgis_analysis::control::Control>) -> CmdResult {
self.invoke_tool_with(op, control, &crate::cmd::gp::RunOpts::default()).map(|(value, _)| value)
}
/// [`Self::invoke_tool`] applied as `opts` says (a batch item), returning the run's messages.
pub(crate) fn invoke_tool_with(
&mut self,
op: Authorized<ToolOp>,
control: &Arc<rustgis_analysis::control::Control>,
opts: &crate::cmd::gp::RunOpts,
) -> Result<(Value, Vec<String>), String> {
let node = op.node.clone();
let number = node.number;
let result = self.with_node(node, |s| crate::cmd::gp::execute(s, op, control));
if let Ok(value) = &result {
let result = self.with_node(node, |s| crate::cmd::gp::execute(s, op, control, opts));
if let Ok((value, _)) = &result {
// Engine apply's result identifies the actual output kind. An identically named
// layer must never substitute for a produced standalone table (or vice versa).
let layer = value
Expand Down
5 changes: 5 additions & 0 deletions crates/engine/src/policy/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ pub fn command_effects(spec: &crate::CommandSpec, params: &serde_json::Value) ->
&& let Ok(node) = plan::tool_effect_node(tool, params.get("params").unwrap_or(&serde_json::Value::Null))
{
effects.union(node)
} else if spec.id == "gp.batch"
&& let Some(tool) = params.get("tool").and_then(serde_json::Value::as_str)
&& let Ok(tool) = plan::batch_tool_effects(tool)
{
effects.union(tool)
} else {
effects
}
Expand Down
19 changes: 19 additions & 0 deletions crates/engine/src/policy/plan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -633,6 +633,12 @@ impl<'a> Builder<'a> {
let tool = params.get("tool").and_then(Value::as_str).ok_or("tool is required")?;
n.children.push(self.tool(tool, params.get("params").unwrap_or(&Value::Null))?);
}
// The batch's items aren't expanded here: it declares Files and Automation, and adds
// what its tool reaches beyond a built-in's (a plugin's code, an unreviewed runner).
"gp.batch" => {
let tool = params.get("tool").and_then(Value::as_str).ok_or("tool is required")?;
n.effects = n.effects.union(batch_tool_effects(tool)?);
}
"gp.spatialJoin" => n.children.push(self.tool("spatialJoin", params)?),
"gp.zonalStats" => n.children.push(self.tool("zonalStats", params)?),
"script.run" if self.strict => {
Expand Down Expand Up @@ -1404,6 +1410,19 @@ impl Session {
}
}

/// What a batch's tool reaches beyond a built-in's, without its per-item parameters.
pub(super) fn batch_tool_effects(name: &str) -> Result<Effects, String> {
let pin = ToolPin::resolve(name)?;
let mut e = Effects::NONE;
if pin.plugin.is_some() {
e = e.with(Effect::Plugin);
}
if pin.resolved.source().is_some_and(|s| s != "geolibre" && s != rustgis_plugins::tools::SOURCE) {
e = e.with(Effect::Unreviewed);
}
Ok(e)
}

/// Projects pinned tool effects for compatibility consumers; execution still uses its own token.
pub(super) fn tool_effect_node(name: &str, params: &Value) -> Result<Effects, String> {
let pin = ToolPin::resolve(name)?;
Expand Down
34 changes: 34 additions & 0 deletions docs/user-guide/analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,40 @@ measurement, also with a count. An incomplete polygon is measured along its rema
edges, without assuming it contains the center. The overlay tools, Select By Location, Spatial Join, Summarize
Within and topology validation refuse latitudes past ±90° before doing their work.

### Batch geoprocessing (`gp.batch`)
Comment thread
quanguet0409 marked this conversation as resolved.
Resolved

Run one tool over many layers in the project or a folder of vector files:

```json
gp.batch={"tool":"buffer","inputs":["Cities","Rivers"],"outputPattern":"{input}_buf","params":{"distance":50,"units":"Kilometers"}}
```

- `"tool"`: the tool's id (such as `"buffer"`, `"simplify"` or `"convexHull"`).
- `"inputs"`: layer names or, on desktop and the command line, file paths; or `"*"` for every
vector layer on the active map, hidden ones included (raster and graphics layers are left out,
with a notice). In the browser, add a file as a layer first and use its layer name.
- `"folder"`: on desktop and the command line, a folder whose `.shp`, `.geojson`, `.json`,
`.gpkg`, `.fgb`, `.parquet`, `.geoparquet`, `.kml`, `.kmz`, `.gpx` and `.csv` files are run over
in alphabetical order. Subfolders and symbolic links are skipped, with a count.
- `"outputPattern"` (or `"output_pattern"`): the name of each output layer, `"{input}_{tool}"` by
default, where `{input}` is the layer's name or the file's name without its extension and
`{tool}` the tool's id. It must include `{input}`. A name already taken gets a number (`_2`).
Tools that change a layer in place or select features keep their own names.
- `"params"`: the parameters every run shares. The batch input replaces a value given there for
the input parameter, with a notice.
- `"param"` (or `"batch_param"`): the parameter each input goes to; by default the tool's first
layer parameter, else `"input"`.

One input failing doesn't stop the others. The result counts what `"succeeded"` and `"failed"`,
names the `"outputs"` (layers or tables) and lists each failure in `"errors"`, with one notice
for the failures. If every input fails, the batch is an error naming each input and why. What a
reader drops or assumes for a file input (Z values, a coordinate system) is one notice per kind,
naming the inputs, and is kept in the history and the result's `"warnings"`. Tools that write a
file are refused. A batch is yours to run: the AI Assistant asks each time, and plugins and
imported tasks can't run one.

### Built-in tools

These are the built-in tools; GeoLibre and plugin tools join them when available.
Use these ids in `gp.run`'s `tool` parameter (`gp.tools` lists the available tools and their parameters):

Expand Down
Loading