Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 29 additions & 18 deletions crates/engine/src/cmd/imagery.rs
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ pub fn commands() -> Vec<C> {
r#"{"layer": raster, "index": "ndvi"|"ndwi"|"savi"|"nbr"|"ndbi"|"evi", "red"?, "green"?, "blue"?, "nir"?, "swir"?: band numbers, "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#,
),
C::new("raster.process", "Process", "Imagery › Imagery Tools", process).inputs(process_inputs).roles(&[]).effects(crate::policy::Effects::NONE)
.params(r#"{"layer": raster, "operation": "clip"|"resample", "extent"?: [minLon, minLat, maxLon, maxLat] (clip; the view when omitted), "polygon"?: polygon layer (clip to its selected or all polygons; cells outside become no-data), "selectedOnly"?: bool (clip to selected polygons when true, all when false), "factor"?: n (resample: n×n pixels become one), "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#),
.params(r#"{"layer": raster, "operation": "clip"|"resample", "extent"?: [minLon, minLat, maxLon, maxLat] (clip; the view when omitted), "polygon"?: polygon layer (clip to its selected or all polygons; cells outside become no-data, or see-through in a raster with alpha or a three-band colour one without no-data), "selectedOnly"?: bool (clip to selected polygons when true, all when false), "factor"?: n (resample: n×n pixels become one), "name"?: output, "ran"?: (internal: a raster the app has worked out)}"#),
C::new("raster.mosaic", "Mosaic Rasters", "Imagery › Imagery Tools", raster_mosaic).inputs(mosaic_inputs).roles(&[]).effects(crate::policy::Effects::NONE.with(crate::policy::Effect::Files)).params(
r#"{"inputs": [raster layer or file path, …] (two or more, in one coordinate system and cell size; where they overlap the first wins), "name"?: output, "path"?: GeoTIFF to write the mosaic to}"#,
),
Expand Down Expand Up @@ -1129,7 +1129,8 @@ pub(crate) fn index_work(s: &Session, p: &Value) -> Result<super::workflows::Wor
/// even-odd rule within each polygon. Where multiple polygons overlap, their coverage is combined
/// (union: any cell falling inside at least one polygon is retained).
/// - The raster is cropped to the tight bounding box of valid selected polygons, and cells outside the
/// polygons receive NoData.
/// polygons receive NoData; in a raster with alpha, or a three-band colour one without NoData (which
/// gains an alpha band), they are see-through, 0 in every band, so its pixel type is kept.
/// - **Partial Failure Reporting**:
/// - Features with invalid geometry are skipped and reported in the result messages.
/// - Polygon parts that cannot be projected onto the raster grid are reported separately.
Expand Down Expand Up @@ -1548,7 +1549,11 @@ fn mosaic_matches(first: &Raster, r: &Raster) -> Result<(), String> {
}

/// The cells of `r` in a column and row window, as a new raster on the same grid. With a mask
/// (row-major over the window), the cells it leaves out become no-data.
/// (row-major over the window), the cells it leaves out have no value, as Project Raster leaves
/// the cells outside its source (`rustgis_analysis::warp`): in a raster with alpha they are
/// see-through, 0 in every band, and a three-band colour raster with neither alpha nor no-data gains an alpha
/// band for them, so an 8-bit image stays 8-bit (#1731). Otherwise they are its no-data, or
/// -3.4e38 when it has none.
fn clip_window(
r: &Raster,
cols: std::ops::Range<u32>,
Expand All @@ -1558,11 +1563,17 @@ fn clip_window(
) -> Result<Raster, String> {
r.require_coordinates()?;
let (w, h) = (cols.end.saturating_sub(cols.start), rows.end.saturating_sub(rows.start));
let nodata = if mask.is_some() { r.nodata.or(Some(NODATA as f64)) } else { r.nodata };
let fill = nodata.map(|v| v as f32).unwrap_or(NODATA);
let n = r.bands.len();
let has_alpha = matches!(r.color, ColorHint::Rgb { alpha: true } if n >= 4) || matches!(r.color, ColorHint::GrayAlpha if n >= 2);
Comment thread
giswqs marked this conversation as resolved.
Resolved
let add_alpha = mask.is_some() && !has_alpha && matches!(r.color, ColorHint::Rgb { .. }) && n == 3 && r.nodata.is_none();
Comment thread
giswqs marked this conversation as resolved.
Resolved
Comment thread
giswqs marked this conversation as resolved.
Resolved
Comment thread
giswqs marked this conversation as resolved.
Resolved
let clear = mask.is_some() && (has_alpha || add_alpha);
let nodata = if mask.is_some() && !clear { r.nodata.or(Some(NODATA as f64)) } else { r.nodata };
let fill = if clear { 0.0 } else { nodata.map(|v| v as f32).unwrap_or(NODATA) };
Comment thread
giswqs marked this conversation as resolved.
Resolved

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reopening #1810's concern with new evidence (medium confidence, behaviour change): the reply to the earlier thread says the see-through cells are "counted the way the app counts them in any RGBA raster", which is true for a raster read from a file. It doesn't cover what this PR changes. Before this PR, a polygon clip's outside cells were no-data (−3.4e38 in a Float32 output, or no-data on an RGB), so BandStats, the stretch and Change Detection excluded them. Now fill = 0.0 makes those cells valid zeros for every clipped RGB and RGBA output, and the result messages don't say so. That is a new way for clipped outputs to change statistics, and "Never silent" asks for it to be stated. Either keep the no-data for the clip output's statistics, or at least say in the command's result messages and the Analysis guide that clipped outside cells are valid zeros with alpha 0 for statistics and Change Detection until #1810 lands.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this as it is. The outside cells are not dropped data: they are see-through, which is what the alpha band says, and the Analysis guide states it ("see-through instead (0, with alpha 0)"). That is the same convention Project Raster already uses for the cells it has no value for, so a clip and a projection of one image now agree. Keeping no-data for them is what widened the 8-bit image to Float32 in #1731, so it would undo the fix. Statistics and Change Detection that ignore alpha-0 cells for every RGBA raster are tracked in #1810. Leaving this thread open for the maintainer.

// Each band's cells counted and cancellable, as in a run on another thread.
let (wu, width) = (w as usize, r.width as usize);
let bands = r
let kept = |i: usize| mask.is_none_or(|m| m.get(i).copied().unwrap_or(false));
let at = |i: usize| (rows.start as usize + i / wu.max(1)) * width + cols.start as usize + i % wu.max(1);
let mut bands = r
.bands
.iter()
.map(|b| {
Expand All @@ -1571,23 +1582,23 @@ fn clip_window(
wu,
1,
|| (),
|_, i| match mask.map(|m| m.get(i).copied().unwrap_or(false)) {
Some(false) => fill,
_ => {
let (row, col) = (rows.start as usize + i / wu.max(1), cols.start as usize + i % wu.max(1));
b.data.get(row * width + col).copied().unwrap_or(fill)
}
},
|_, i| if kept(i) { b.data.get(at(i)).copied().unwrap_or(fill) } else { fill },
Comment thread
giswqs marked this conversation as resolved.
Resolved
)
})
.collect::<Result<Vec<_>, _>>()?;
if add_alpha {
// Opaque where the cell is kept and has a value in some band, as Project Raster's alpha.
let valued = |i: usize| r.bands.iter().any(|b| b.data.get(at(i)).is_some_and(|v| !v.is_nan()));
Comment thread
giswqs marked this conversation as resolved.
Resolved
bands.push(rasterops::per_index(wu * h as usize, wu, n, || (), |_, i| if kept(i) && valued(i) { 255.0 } else { 0.0 })?);
}
let o = r.transform.to_world(f64::from(cols.start), f64::from(rows.start));
let gt = GeoTransform { origin_x: o.x, origin_y: o.y, ..r.transform };
let mut out = Raster::new(name, w, h, bands, gt, r.crs, nodata, r.color.clone(), r.data_type.clone()).map_err(|e| e.to_string())?;
out.bands.iter_mut().zip(&r.bands).for_each(|(o, i)| {
o.name = i.name.clone();
o.wavelength = i.wavelength;
});
let color = if add_alpha { ColorHint::Rgb { alpha: true } } else { r.color.clone() };
let mut out = Raster::new(name, w, h, bands, gt, r.crs, nodata, color, r.data_type.clone()).map_err(|e| e.to_string())?;
for (i, band) in out.bands.iter_mut().enumerate() {
band.name = r.bands.get(i).map_or_else(|| "Alpha".to_string(), |b| b.name.clone());
band.wavelength = r.bands.get(i).and_then(|b| b.wavelength);
}
// Some of the values rounded when the file was read may be in the window: how many isn't known.
out.keep_rounded_mark(r);
Ok(out)
Expand Down
119 changes: 119 additions & 0 deletions crates/engine/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -41595,6 +41595,125 @@ fn a_model_trained_within_the_work_limit_says_so() {
assert_eq!(r["epochs"], 3, "{r}");
}

/// A grey image with alpha clipped to a polygon stays 8-bit with two bands: the cells outside are
/// see-through, 0 in both bands, and the cells inside keep their grey and alpha (#1731).
#[test]
fn a_clipped_grey_raster_with_alpha_stays_8_bit() {
use rustgis_data::raster::{ColorHint, Crs, GeoTransform, Raster, geotiff};
let utm = Crs::from_epsg(32611).unwrap();
let (w, h) = (40u32, 40u32);
let n = (w * h) as usize;
let gt = GeoTransform { origin_x: 500_000.0, origin_y: 4_800_040.0, pixel_width: 1.0, pixel_height: -1.0 };
let grey = |i: usize| (10 + (i * 7) % 240) as f32;
let bands = vec![(0..n).map(grey).collect(), vec![255.0; n]];
let r = Raster::new("grey", w, h, bands, gt, utm, None, ColorHint::GrayAlpha, "UInt8").unwrap();
let mut s = Session::new(Project::default());
let bytes = geotiff::write(&r, 0).unwrap();
crate::cmd::raster::add_raster(&mut s, "grey", rustgis_map::RasterLayer::from_bytes(r, "grey.tif", bytes)).unwrap();
let tr = utm.transformer().unwrap();
let ll = |x: f64, y: f64| {
let c = tr.to_lonlat(rustgis_geom::Coord::new(x, y)).unwrap();
json!([c.x, c.y])
};
// The same triangle as in `a_clipped_colour_raster_stays_8_bit`.
s.run("layer.template", &json!({"template": "polygons", "name": "Boundary"})).unwrap();
let ring = [ll(500_005.0, 4_800_035.0), ll(500_035.0, 4_800_035.0), ll(500_035.0, 4_800_005.0), ll(500_005.0, 4_800_035.0)];
s.run("edit.addFeature", &json!({"layer": "Boundary", "geometry": {"type": "Polygon", "coordinates": [ring]}})).unwrap();
s.run(
"raster.process",
&json!({"layer": "grey", "operation": "clip", "polygon": "Boundary", "selectedOnly": false, "name": "clipped"}),
)
.unwrap();
let c = s.map().layer_by_name("clipped").unwrap().raster.as_ref().unwrap().data.clone().unwrap();
assert_eq!((c.data_type.as_str(), c.nodata, c.bands.len()), ("UInt8", None, 2));
assert_eq!(c.color, ColorHint::GrayAlpha);
let cell = |row: u32, col: u32| c.bands.iter().map(|b| b.data[(row * c.width + col) as usize]).collect::<Vec<f32>>();
assert_eq!(cell(c.height - 1, 0), vec![0.0, 0.0]);
// The window starts 5 cells in: its upper right is the source's row 5, column 34.
assert_eq!(cell(0, c.width - 1), vec![grey(5 * w as usize + 34), 255.0]);
}

/// An 8-bit colour orthophoto clipped to a polygon, as it is or projected (bilinear) first, stays
/// 8-bit, and its exports too, as a COG and a plain GeoTIFF: the cells outside the polygon are
/// see-through, 0 in every band, as Project Raster leaves the cells it has no value for. An RGB
/// image without no-data gains an alpha band for them; one with a no-data value bytes hold keeps it
/// and its three bands (#1731).
#[test]
fn a_clipped_colour_raster_stays_8_bit() {
use rustgis_data::raster::{ColorHint, Crs, GeoTransform, Raster, geotiff};
let utm = Crs::from_epsg(32611).unwrap();
let (w, h) = (40u32, 40u32);
let n = (w * h) as usize;
let gt = GeoTransform { origin_x: 500_000.0, origin_y: 4_800_040.0, pixel_width: 1.0, pixel_height: -1.0 };
let dir = std::env::temp_dir().join(format!("rustgis-clip-colour-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
// Colours 10–249, opaque.
let value = |i: usize, b: usize| if b == 3 { 255.0 } else { (10 + (i * 7 + b * 60) % 240) as f32 };
for (label, nb, color, nodata) in [
("rgba", 4usize, ColorHint::Rgb { alpha: true }, None),
("rgb", 3, ColorHint::Rgb { alpha: false }, None),
("rgb with no-data", 3, ColorHint::Rgb { alpha: false }, Some(0.0)),
] {
let bands: Vec<Vec<f32>> = (0..nb).map(|b| (0..n).map(|i| value(i, b)).collect()).collect();
let r = Raster::new("ortho", w, h, bands, gt, utm, nodata, color, "UInt8").unwrap();
let mut s = Session::new(Project::default());
let bytes = geotiff::write(&r, 0).unwrap();
crate::cmd::raster::add_raster(&mut s, "ortho", rustgis_map::RasterLayer::from_bytes(r, "ortho.tif", bytes)).unwrap();
let tr = utm.transformer().unwrap();
let ll = |x: f64, y: f64| {
let c = tr.to_lonlat(rustgis_geom::Coord::new(x, y)).unwrap();
json!([c.x, c.y])
};
// A triangle over the upper right of a 30 m square: cells on and above its diagonal.
s.run("layer.template", &json!({"template": "polygons", "name": "Boundary"})).unwrap();
let ring = [ll(500_005.0, 4_800_035.0), ll(500_035.0, 4_800_035.0), ll(500_035.0, 4_800_005.0), ll(500_005.0, 4_800_035.0)];
s.run("edit.addFeature", &json!({"layer": "Boundary", "geometry": {"type": "Polygon", "coordinates": [ring]}})).unwrap();
s.run("raster.project", &json!({"layer": "ortho", "crs": 32611, "cellSize": 0.5, "resampling": "bilinear", "name": "projected"}))
.unwrap();
for (from, to) in [("ortho", "clipped"), ("projected", "projected clipped")] {
s.run("raster.process", &json!({"layer": from, "operation": "clip", "polygon": "Boundary", "selectedOnly": false, "name": to}))
.unwrap();
}
// Project Raster adds alpha to an RGB image without no-data, as Clip now does.
let (bands_out, nodata_out) = if nodata.is_some() { (3, nodata) } else { (4, None) };
for name in ["clipped", "projected clipped"] {
let r = s.map().layer_by_name(name).unwrap().raster.as_ref().unwrap().data.clone().unwrap();
let what = format!("{label} {name}");
assert_eq!((r.data_type.as_str(), r.nodata, r.bands.len()), ("UInt8", nodata_out, bands_out), "{what}");
assert_eq!(r.color, ColorHint::Rgb { alpha: bands_out == 4 }, "{what}");
// The lower left corner is outside the triangle: 0 in every band. The upper right is
// inside: opaque.
let cell = |row: u32, col: u32| r.bands.iter().map(|b| b.data[(row * r.width + col) as usize]).collect::<Vec<f32>>();
assert_eq!(cell(r.height - 1, 0), vec![0.0; bands_out], "{what}");
let inside = cell(0, r.width - 1);
assert!(inside.iter().take(3).all(|v| (10.0..=249.0).contains(v)), "{what}: {inside:?}");
if bands_out == 4 {
assert_eq!(inside[3], 255.0, "{what}");
}
if bands_out == 4 && nb == 3 {
assert_eq!(r.bands[3].name, "Alpha", "{what}");
}
if name == "clipped" {
// The window starts 5 cells in: its upper right is the source's row 5, column 34.
let i = 5 * w as usize + 34;
assert_eq!(
inside.iter().take(3).copied().collect::<Vec<f32>>(),
(0..3).map(|b| value(i, b)).collect::<Vec<f32>>(),
"{what}"
);
}
for cog in [true, false] {
let path = dir.join(format!("{}-{cog}.tif", what.replace(' ', "_")));
s.run("share.exportRaster", &json!({"layer": name, "path": path.display().to_string(), "cog": cog})).unwrap();
let back = geotiff::read(std::io::Cursor::new(std::fs::read(&path).unwrap()), "back").unwrap();
assert_eq!((back.data_type.as_str(), back.nodata, back.bands.len()), ("UInt8", nodata_out, bands_out), "{what} cog={cog}");
assert!(back.bands.iter().all(|b| b.data.iter().all(|v| (0.0..=255.0).contains(v))), "{what} cog={cog}");
}
}
}
std::fs::remove_dir_all(dir).unwrap();
}

/// A STAC search limited to a shape (`intersects`, from GeoJSON or the selected features, widened
/// by `buffer` metres): the shape is sent in place of the bbox, what merging leaves out is said
/// once, and clean input says nothing.
Expand Down
9 changes: 8 additions & 1 deletion docs/user-guide/analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -1503,7 +1503,14 @@ The Imagery tab's groups open their tools in the Workflows pane. The tab belongs
- **Change Detection** (difference, ratio, percent), **Indices** (NDVI, NDWI, SAVI, NBR, NDBI,
EVI), **Process** (clip to the view, to a box, or to a polygon layer's selected or all polygons,
with the cells outside them no-data; resample) and the **Pixel Editor** (set the pixels inside a
polygon).
polygon). As in Project Raster, the cells outside the polygons are see-through instead (0, with
alpha 0) in an image with an alpha band, and a three-band colour image with neither alpha nor
no-data gains one for them, so an 8-bit image stays 8-bit when it is exported. A colour image
with a fourth band of data and no alpha (RGB and near-infrared) gets no-data there as before
(−3.4e38 when it has none). Project Raster doesn't give it an alpha band either: a colour image's
alpha is read from its fourth band, which holds its data. A clip that gained an alpha band has
four bands, so Mosaic Rasters refuses it beside three-band tiles, saying how many bands each
has: clip the others the same way, or mosaic first and clip the result.
- **Mosaic Rasters** (`raster.mosaic`) merges tiles of one grid, such as the 1 km tiles of a state
elevation model, into one raster in their own coordinate system and cell size, optionally
written to a GeoTIFF: `{"inputs": ["dgm_1.tif", "dgm_2.tif", …], "path": "dem.tif"}`. Inputs
Expand Down
Loading