Repository navigation
fix(data): a large GeoTIFF opens from an overview whose size was rounded down, not by reading its whole full image (#1965) - #2391
Conversation
…ded down, not by reading its whole full image (opengeos#1965)
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
| let as_detailed = |pixels: u64| pixels >= u64::from((full_w / k).max(1)) * u64::from((full_h / k).max(1)); | ||
| match best { | ||
| // An overview at least as detailed as the full image read sparsely: only it is read. | ||
| Some((index, ow, oh)) if decimated.is_none_or(|d| u64::from(ow) * u64::from(oh) >= d) => { | ||
| // An overview as detailed, to the rounding of its size, as the full image read | ||
| // sparsely: only it is read. | ||
| Some((index, ow, oh)) if decimated.is_none() || as_detailed(u64::from(ow) * u64::from(oh)) => { |
There was a problem hiding this comment.
Low confidence (~35%), behaviour change to confirm: the new comment says an overview counts as "as detailed" when it is the full size over k "rounded either way", which is a per-axis rule. The closure compares areas (ow * oh >= floor_w * floor_h), so an overview that is short in one axis can pass if it is long in the other. For example, a 10,000 × 10,300 overview of 83517 × 78739 over 8 has area 103.0M, above the 102.7M floor product, and is accepted even though its width is 439 pixels short. The old ceil-area rule had the same shape, but this change widens the window, so the per-axis check is the stricter and clearer way to state the rule. Real writers produce proportional overviews, so this may never happen in practice. If you agree, the suggestion below also fixes the match guard that uses the closure:
| let as_detailed = |pixels: u64| pixels >= u64::from((full_w / k).max(1)) * u64::from((full_h / k).max(1)); | |
| match best { | |
| // An overview at least as detailed as the full image read sparsely: only it is read. | |
| Some((index, ow, oh)) if decimated.is_none_or(|d| u64::from(ow) * u64::from(oh) >= d) => { | |
| // An overview as detailed, to the rounding of its size, as the full image read | |
| // sparsely: only it is read. | |
| Some((index, ow, oh)) if decimated.is_none() || as_detailed(u64::from(ow) * u64::from(oh)) => { | |
| let as_detailed = |ow: u32, oh: u32| ow >= (full_w / k).max(1) && oh >= (full_h / k).max(1); | |
| match best { | |
| // An overview as detailed, to the rounding of its size, as the full image read | |
| // sparsely: only it is read. | |
| Some((index, ow, oh)) if decimated.is_none() || as_detailed(ow, oh) => { |
Left for the release quality pass (not raised inline): decimated is now used only for is_none(), so a plain let decimable = k <= 1 << 20; would read better, and the floor values are computed twice (here and in dw/dh).
There was a problem hiding this comment.
Thanks. Left as it is: the old rule was already an area comparison (ceil × ceil), so a disproportionate overview could pass it before too; this only moves its threshold to the floor product, at most one row and one column of the decimated grid. A per-axis rule alone would refuse overviews the area rule accepted until now (long one way, short the other), and those files would go back to reading their whole full image, the slowdown this PR fixes. Accepting either rule would cover both, but no writer makes such overviews (GDAL rounds up, terrio down, both proportionally), so it isn't worth the extra condition here.
Code reviewI found no correctness bugs in the change. The overview is read with the right per-axis scale, and the new rule keeps the old behaviour for rounded-up files. One low-confidence design point is posted inline. I did not run the tests here, because this session has no shell. Everything below is from reading the diff and the surrounding code. Bugs
Security
Performance
Quality (low confidence unless noted; left for the release quality pass)
AGENTS.md
Prior concerns: none. The review-thread file was empty, so there was nothing to deduplicate. Inline comments posted: one, on Change sizeCompared with main: the whole PR (merge base 99c9f31 to head 4803ae2).
|
📖 Docs preview
GitHub Pages takes a minute or two to publish a new commit. |
Summary
Fixes #1965: a 10 GB cloud-optimised mosaic took minutes to open on the desktop, though it holds every overview it needs.
Cause. A local GeoTIFF too large to hold whole opens either from its largest overview that fits the memory limit, or by reading the full image and keeping every k-th pixel. The overview is used when it is at least as detailed as that every-k-th-pixel read. The reporter's file is 83517 × 78739 pixels (3 bands, 512-pixel LZW tiles, eight internal overviews), written by
terrio mosaic --format=cog, which rounds each overview's size down: its 1:8 overview is 10439 × 9842. Every 8th pixel of the full image, rounded up as RustGIS sized it, is 10440 × 9843. One pixel short in each direction, the overview lost, and the whole 10.5 GB full image was read to keep every 8th pixel.This applies on any computer with 16 GB of memory or more, where the limit allows every 8th pixel. GDAL rounds overview sizes up, so files it writes weren't affected, which is why a synthetic COG of even size opened in seconds.
Fix. An overview counts as being as detailed as every k-th pixel when it is the full size over k rounded either way. One that is really coarser still loses, and the full image is still read sparsely then (#162's 5000-pixel orthophoto with a 157-pixel overview).
Measured on the reporter's file (its header and the overviews RustGIS reads, fetched by byte range: 184 MB of the 10.5 GB), Linux,
rustgis-cli run --threads 6 --data MOSAICO_cog.tif:mainThe notice is unchanged: "showing a 10439 × 9842 overview of the 83517 × 78739 raster".
Surfaces
read_sparse), so it now also fetches only the overview for such a file.layer.addUrlchoose levels by pixel size, not by this comparison, so they are unchanged.Tests
geotiff::tests::an_overview_rounded_down_is_read_rather_than_the_full_image: a 1001-pixel image with overviews of 500 and 250 (rounded down, as terrio writes them), at a limit where every 2nd pixel is 501 × 501. The 500 overview is read. Read a part at a time, the reader holds under a third of the file, so none of the full image's tiles. A file rounded up as GDAL writes it reads its 501 overview as before. The test fails onmain(it reads 501 × 501 from the full image).cog::Examplegainsrounded_downfor that fixture.cargo xtask changed(fmt, clippy, the changed packages' tests). Only the three failures known on this machine (filegdbwith GDAL installed, two shapefile tests) fail, as onmain.terrio's overview sizes are mentioned to terrio's maintainers separately; RustGIS reads either rounding.
Summary by CodeRabbit