Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion crates/data/src/raster/cog.rs
Original file line number Diff line number Diff line change
Expand Up @@ -748,6 +748,9 @@ pub struct Example {
/// Without `classes`: band `b`'s wavelength, 400 + 100 b nanometres, in GDAL's metadata
/// (its only item there).
pub wavelengths: bool,
/// Overviews half the size before them rounded down (500 of 1001), as terrio 0.8 writes
/// them; GDAL rounds up (501).
pub rounded_down: bool,
}

impl Default for Example {
Expand All @@ -770,6 +773,7 @@ impl Default for Example {
sparse: false,
no_overviews: false,
wavelengths: false,
rounded_down: false,
}
}
}
Expand All @@ -783,7 +787,8 @@ pub fn example_with(e: &Example) -> Vec<u8> {
let per_chunk = nb / planes;
let mut sizes = vec![size];
while !e.no_overviews && *sizes.last().unwrap_or(&tile) > tile {
sizes.push(sizes.last().unwrap_or(&tile).div_ceil(2));
let last = *sizes.last().unwrap_or(&tile);
sizes.push(if e.rounded_down { last / 2 } else { last.div_ceil(2) });
}
let value = |x: u32, y: u32, b: u32| -> u8 {
if e.int8 {
Expand Down
32 changes: 30 additions & 2 deletions crates/data/src/raster/geotiff.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1430,9 +1430,14 @@ pub(super) fn read_limited_with_nodata(
}
let (dw, dh) = (full_w.div_ceil(k), full_h.div_ceil(k));
let decimated = (k <= 1 << 20).then_some(u64::from(dw) * u64::from(dh));
// As detailed as every k-th pixel is an overview of the full size over k, rounded either
// way: GDAL rounds up, some writers down (terrio's 10439 × 9842 for 83517 × 78739 over
// 8, #1965), and one a pixel short must not make the whole full image be read instead.
let as_detailed = |pixels: u64| pixels >= u64::from((full_w / k).max(1)) * u64::from((full_h / k).max(1));
match best {
// An overview at least as detailed as the full image read sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none_or(|d| u64::from(ow) * u64::from(oh) >= d) => {
// An overview as detailed, to the rounding of its size, as the full image read
// sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none() || as_detailed(u64::from(ow) * u64::from(oh)) => {
Comment on lines +1436 to +1440

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low confidence (~35%), behaviour change to confirm: the new comment says an overview counts as "as detailed" when it is the full size over k "rounded either way", which is a per-axis rule. The closure compares areas (ow * oh >= floor_w * floor_h), so an overview that is short in one axis can pass if it is long in the other. For example, a 10,000 × 10,300 overview of 83517 × 78739 over 8 has area 103.0M, above the 102.7M floor product, and is accepted even though its width is 439 pixels short. The old ceil-area rule had the same shape, but this change widens the window, so the per-axis check is the stricter and clearer way to state the rule. Real writers produce proportional overviews, so this may never happen in practice. If you agree, the suggestion below also fixes the match guard that uses the closure:

Suggested change
let as_detailed = |pixels: u64| pixels >= u64::from((full_w / k).max(1)) * u64::from((full_h / k).max(1));
match best {
// An overview at least as detailed as the full image read sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none_or(|d| u64::from(ow) * u64::from(oh) >= d) => {
// An overview as detailed, to the rounding of its size, as the full image read
// sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none() || as_detailed(u64::from(ow) * u64::from(oh)) => {
let as_detailed = |ow: u32, oh: u32| ow >= (full_w / k).max(1) && oh >= (full_h / k).max(1);
match best {
// An overview as detailed, to the rounding of its size, as the full image read
// sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none() || as_detailed(ow, oh) => {

Left for the release quality pass (not raised inline): decimated is now used only for is_none(), so a plain let decimable = k <= 1 << 20; would read better, and the floor values are computed twice (here and in dw/dh).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Left as it is: the old rule was already an area comparison (ceil × ceil), so a disproportionate overview could pass it before too; this only moves its threshold to the floor product, at most one row and one column of the decimated grid. A per-axis rule alone would refuse overviews the area rule accepted until now (long one way, short the other), and those files would go back to reading their whole full image, the slowdown this PR fixes. Accepting either rule would cover both, but no writer makes such overviews (GDAL rounds up, terrio down, both proportionally), so it isn't worth the extra condition here.

dec.seek_to_image(index).map_err(err)?;
image = index;
(w, h) = (ow, oh);
Expand Down Expand Up @@ -2922,6 +2927,29 @@ mod tests {
assert_eq!(back.value(0, 517, 300), r.value(0, 517, 300));
}

/// #1965: an overview a pixel short of every k-th pixel of the full image, as a writer that
/// rounds its overviews' sizes down makes it (terrio's 10439 × 9842 for 83517 × 78739 over 8),
/// is read rather than the whole full image, whose every k-th pixel is only a pixel larger. A
/// 10 GB mosaic took minutes to open so.
#[test]
fn an_overview_rounded_down_is_read_rather_than_the_full_image() {
use super::super::cog::{Example, example_with};
// 1001 pixels a side in 256-pixel tiles, overviews of 500 and 250. At most 260,000
// samples: every second pixel is 501 × 501, the 500 overview a pixel short each way.
let file = example_with(&Example { size: 1001, tile: 256, rounded_down: true, ..Example::default() });
let max = 260_000;
let whole = read_limited(reader(std::io::Cursor::new(&file)).unwrap(), "rounded", max, None).unwrap();
assert_eq!((whole.width, whole.height, whole.overview), (500, 500, Some((1001, 1001))));
// Only the overview's tiles are read, none of the full image's (most of the file).
let (r, parts) = read_in_parts(&file, 64 << 10, max);
assert_eq!((r.width, r.height), (500, 500));
assert!(parts.held() < file.len() as u64 / 3, "{} of {}", parts.held(), file.len());
// One rounded up, as GDAL writes it, is read as before.
let up = example_with(&Example { size: 1001, tile: 256, ..Example::default() });
let r = read_limited(reader(std::io::Cursor::new(&up)).unwrap(), "up", max, None).unwrap();
assert_eq!((r.width, r.height), (501, 501));
}

#[test]
fn a_striped_file_read_in_parts_is_read_to_its_end() {
// No overviews and in strips: every strip is needed, and reads stop at the file's end.
Expand Down
Loading