Skip to content

fix(data): a large GeoTIFF opens from an overview whose size was rounded down, not by reading its whole full image (#1965) - #2391

Merged
pick2510 merged 1 commit into
opengeos:mainfrom
pick2510:fix/local-cog-open
Nov 18, 2026
Merged

pick2510 merged 1 commit into
opengeos:mainfrom
pick2510:fix/local-cog-open

Conversation

@pick2510

@pick2510 pick2510 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #1965: a 10 GB cloud-optimised mosaic took minutes to open on the desktop, though it holds every overview it needs.

Cause. A local GeoTIFF too large to hold whole opens either from its largest overview that fits the memory limit, or by reading the full image and keeping every k-th pixel. The overview is used when it is at least as detailed as that every-k-th-pixel read. The reporter's file is 83517 × 78739 pixels (3 bands, 512-pixel LZW tiles, eight internal overviews), written by terrio mosaic --format=cog, which rounds each overview's size down: its 1:8 overview is 10439 × 9842. Every 8th pixel of the full image, rounded up as RustGIS sized it, is 10440 × 9843. One pixel short in each direction, the overview lost, and the whole 10.5 GB full image was read to keep every 8th pixel.

This applies on any computer with 16 GB of memory or more, where the limit allows every 8th pixel. GDAL rounds overview sizes up, so files it writes weren't affected, which is why a synthetic COG of even size opened in seconds.

Fix. An overview counts as being as detailed as every k-th pixel when it is the full size over k rounded either way. One that is really coarser still loses, and the full image is still read sparsely then (#162's 5000-pixel orthophoto with a 157-pixel overview).

Measured on the reporter's file (its header and the overviews RustGIS reads, fetched by byte range: 184 MB of the 10.5 GB), Linux, rustgis-cli run --threads 6 --data MOSAICO_cog.tif:

Opens from Reads
0.6.0 and main every 8th pixel of the full image the whole 10.5 GB
This PR the 10439 × 9842 overview 184 MB, in 4.1 s

The notice is unchanged: "showing a 10439 × 9842 overview of the 83517 × 78739 raster".

Surfaces

  • Desktop app, command line and Python: the local GeoTIFF read is this path.
  • The browser reads large GeoTIFFs a part at a time through the same function (read_sparse), so it now also fetches only the overview for such a file.
  • Phone apps: the same reader.
  • Tile layers and layer.addUrl choose levels by pixel size, not by this comparison, so they are unchanged.
  • No command, setting or saved field changes. Nothing to undo.

Tests

  • geotiff::tests::an_overview_rounded_down_is_read_rather_than_the_full_image: a 1001-pixel image with overviews of 500 and 250 (rounded down, as terrio writes them), at a limit where every 2nd pixel is 501 × 501. The 500 overview is read. Read a part at a time, the reader holds under a third of the file, so none of the full image's tiles. A file rounded up as GDAL writes it reads its 501 overview as before. The test fails on main (it reads 501 × 501 from the full image).
  • cog::Example gains rounded_down for that fixture.
  • Locally: cargo xtask changed (fmt, clippy, the changed packages' tests). Only the three failures known on this machine (filegdb with GDAL installed, two shapefile tests) fail, as on main.

terrio's overview sizes are mentioned to terrio's maintainers separately; RustGIS reads either rounding.

Summary by CodeRabbit

  • Bug Fixes
    • Raster images with dimensions rounded down for overviews now select the appropriate overview during decimated reads, including for odd-sized images.
    • Sparse reads can use the selected overview data instead of fetching full-resolution image data unnecessarily.
  • New Features
    • Overview generation can now use dimensions rounded down at each level, while retaining the existing rounding behavior by default.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change adds an option to generate overview dimensions by rounding down. GeoTIFF overview selection now accepts such dimensions, and a regression test covers selection and sparse reading.

Changes: Overview generation and selection

Layer / File(s) Summary
Rounded-down overview dimensions
crates/data/src/raster/cog.rs
Example adds a rounded_down option, defaulting to false. When enabled, overview dimensions use integer division by two; otherwise, the existing round-up calculation remains.
GeoTIFF overview selection
crates/data/src/raster/geotiff.rs
Overview selection now compares pixel count against the product of floored dimensions. A regression test checks that a 500×500 overview is selected for a 1001×1001 image, sparse reads use less than one third of the file, and a 501×501 overview remains selectable.

Estimated code review effort: 2 (Simple) | ~15 minutes

Change: Bug fix · Severity of issue fixed: Medium

Possibly related issues

  • opengeos/RustGIS#1227 — This PR changes GeoTIFF overview selection, but does not address the issue’s separate gap in zoom-based overview selection.

Possibly related PRs

  • opengeos/RustGIS#1926 — Both PRs change overview acceptance rules, and their dimension checks must remain consistent.

Suggested reviewers: giswqs, elpidioassinaturas

Merge Risk: Minimal · up to 4803a

This change lets large GeoTIFFs with rounded-down overviews open from the overview instead of reading the full image. No actionable merge-blocking risk was found.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: a large GeoTIFF opens from a rounded-down overview instead of reading the full image. It is specific and matches the raw summary, the description, and the objectives. …
Description check Passed The description follows the template's Summary, Testing, UI evidence, and Related issue sections. Summary explains cause, fix, and measured results. Testing names the new regression test and the local…
Linked Issues check Passed Issue #1965 reports that a 10 GB COG mosaic takes minutes to open in the desktop app, because RustGIS reads the full image instead of using an overview. The PR targets that cause. The overview-selecti…
Out of Scope Changes check Passed The changes are limited to three areas. In crates/data/src/raster/cog.rs, the Example fixture gains a rounded_down option (default false) used by the new test. In crates/data/src/raster/geotiff.rs, th…
Full details: Title check

Explanation

The title names the main change: a large GeoTIFF opens from a rounded-down overview instead of reading the full image. It is specific and matches the raw summary, the description, and the objectives. It uses a fix(data) prefix and includes the issue number, which is acceptable.

Full details: Description check

Explanation

The description follows the template's Summary, Testing, UI evidence, and Related issue sections. Summary explains cause, fix, and measured results. Testing names the new regression test and the local checks. UI evidence states 'No visible UI change.' as the template requires. Related issue links #1965 in the Summary, not in a Related issue section, which is a minor gap.

Full details: Linked Issues check

Explanation

Issue #1965 reports that a 10 GB COG mosaic takes minutes to open in the desktop app, because RustGIS reads the full image instead of using an overview. The PR targets that cause. The overview-selection change in crates/data/src/raster/geotiff.rs accepts an overview whose dimensions equal the full size divided by the decimation factor, rounded either up or down. The author reports that the reporter's file now opens from its 10439 × 9842 overview, reading 184 MB in 4.1 s instead of the 10.5 GB full image. The regression test geotiff::tests::an_overview_rounded_down_is_read_rather_than_the_full_image covers the rounded-down case, a rounded-up control case, and a sparse-read byte bound. The test is described in the PR and the raw summary. I did not read the overview-selection hunk itself: the head file read was truncated before that code, and the diff read failed because blobs were missing. The assessment relies on the PR description and the raw summary for that code. The issue asks for loading as fast as QGIS. The PR does not claim that every large mosaic opens instantly; it claims the rounded-down overview case, which matches the reported file. No coding requirement is unmet on the evidence available.

Full details: Out of Scope Changes check

Explanation

The changes are limited to three areas. In crates/data/src/raster/cog.rs, the Example fixture gains a rounded_down option (default false) used by the new test. In crates/data/src/raster/geotiff.rs, the overview-selection rule changes and a regression test is added. The PR describes both as serving the rounded-down overview fix. The PR states that the notice text, commands, settings, and tile-layer and layer.addUrl level selection are unchanged. The raw summary reports no changed public entity other than the test-fixture field Example::rounded_down, which is needed by the test. I found no demonstrated unrelated change. The overview-selection hunk was not read directly, so this verdict relies on the PR description and summary.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A rabbit counts the pixels down,
Two halves make overviews abound.
A rounded level finds its place,
Sparse reads hop with lighter pace.
The old round-up still joins the race.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 757fbf85-9695-4284-b736-17484164a99b
Commits

Reviewing files that changed from the base of the PR and between 3e12f42 and 4803ae2.

Files selected for processing: 2

  • crates/data/src/raster/cog.rs
  • crates/data/src/raster/geotiff.rs

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Comment on lines +1436 to +1440
let as_detailed = |pixels: u64| pixels >= u64::from((full_w / k).max(1)) * u64::from((full_h / k).max(1));
match best {
// An overview at least as detailed as the full image read sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none_or(|d| u64::from(ow) * u64::from(oh) >= d) => {
// An overview as detailed, to the rounding of its size, as the full image read
// sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none() || as_detailed(u64::from(ow) * u64::from(oh)) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low confidence (~35%), behaviour change to confirm: the new comment says an overview counts as "as detailed" when it is the full size over k "rounded either way", which is a per-axis rule. The closure compares areas (ow * oh >= floor_w * floor_h), so an overview that is short in one axis can pass if it is long in the other. For example, a 10,000 × 10,300 overview of 83517 × 78739 over 8 has area 103.0M, above the 102.7M floor product, and is accepted even though its width is 439 pixels short. The old ceil-area rule had the same shape, but this change widens the window, so the per-axis check is the stricter and clearer way to state the rule. Real writers produce proportional overviews, so this may never happen in practice. If you agree, the suggestion below also fixes the match guard that uses the closure:

Suggested change
let as_detailed = |pixels: u64| pixels >= u64::from((full_w / k).max(1)) * u64::from((full_h / k).max(1));
match best {
// An overview at least as detailed as the full image read sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none_or(|d| u64::from(ow) * u64::from(oh) >= d) => {
// An overview as detailed, to the rounding of its size, as the full image read
// sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none() || as_detailed(u64::from(ow) * u64::from(oh)) => {
let as_detailed = |ow: u32, oh: u32| ow >= (full_w / k).max(1) && oh >= (full_h / k).max(1);
match best {
// An overview as detailed, to the rounding of its size, as the full image read
// sparsely: only it is read.
Some((index, ow, oh)) if decimated.is_none() || as_detailed(ow, oh) => {

Left for the release quality pass (not raised inline): decimated is now used only for is_none(), so a plain let decimable = k <= 1 << 20; would read better, and the floor values are computed twice (here and in dw/dh).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Left as it is: the old rule was already an area comparison (ceil × ceil), so a disproportionate overview could pass it before too; this only moves its threshold to the floor product, at most one row and one column of the decimated grid. A per-axis rule alone would refuse overviews the area rule accepted until now (long one way, short the other), and those files would go back to reading their whole full image, the slowdown this PR fixes. Accepting either rule would cover both, but no writer makes such overviews (GDAL rounds up, terrio down, both proportionally), so it isn't worth the extra condition here.

@github-actions

Copy link
Copy Markdown
Contributor

Code review

I found no correctness bugs in the change. The overview is read with the right per-axis scale, and the new rule keeps the old behaviour for rounded-up files. One low-confidence design point is posted inline. I did not run the tests here, because this session has no shell. Everything below is from reading the diff and the surrounding code.

Bugs

  • None confirmed. The georeferencing takes full_w / w and full_h / h from the overview's actual size (geotiff.rs:1646-1647), so a one-pixel-short overview does not shift the extent. The k value cannot be zero, since the div_ceil above it would already have panicked. Both new expressions are u64 products of u32 values, so they cannot overflow.

Security

  • None found. The change only alters which image is read, and it adds no input parsing.

Performance

  • None found. The change reads less data for the rounded-down case, which is the point of the PR.

Quality (low confidence unless noted; left for the release quality pass)

  • The decimated variable is now used only for is_none(). A plain k <= 1 << 20 boolean would be clearer, and the floor sizes are computed twice (geotiff.rs:1436 and dw/dh at :1431). Confidence 60%.
  • cog::Example and example_with are public, non-test items (cog.rs:706-783), so the new rounded_down field adds to a public struct. That is consistent with how the fixture already works (#[doc(hidden)]). Confidence 40%.
  • No test pins the boundary where a two-pixel-short overview still loses. The existing coarse-overview case (geotiff.rs:2500-2510) is far from that boundary. Confidence 50%.

AGENTS.md

  • Compliant on the points I checked. No unwrap or expect is added to non-test code (the new unwrap_or calls were already there). The notice wording is unchanged, so the overview is still reported. The fixture covers both rounding directions. The change is not a new background path, so the cancellation lists do not apply. The change is pure Rust with no new std::time use, so the wasm build is not affected. Rustfmt's max_width = 140 is respected.

Prior concerns: none. The review-thread file was empty, so there was nothing to deduplicate.

Inline comments posted: one, on crates/data/src/raster/geotiff.rs:1436-1440, for the area-versus-per-axis point (low confidence, with a suggestion block that also updates the match guard).

Change size

Compared with main: the whole PR (merge base 99c9f31 to head 4803ae2).

Files changed Added Modified Removed Renamed Size before Size after Total change
2 0 2 0 0 292.6 KB 294.9 KB +2.3 KB
Path Size before Size after Size change
crates/data/src/raster/geotiff.rs 232.9 KB 235.0 KB +2.0 KB
crates/data/src/raster/cog.rs 59.8 KB 60.0 KB +256 B

@pick2510
pick2510 enabled auto-merge November 18, 2026 07:40
@github-actions

Copy link
Copy Markdown
Contributor

📖 Docs preview

Item Value
Preview https://preview.rustgis.app/pr-2391/
Web app https://preview.rustgis.app/pr-2391/app/
Commit 4803ae2

GitHub Pages takes a minute or two to publish a new commit.

@pick2510
pick2510 added this pull request to the merge queue Nov 18, 2026
Merged via the queue into opengeos:main with commit 7e9718e Nov 18, 2026
24 checks passed
@pick2510
pick2510 deleted the fix/local-cog-open branch November 18, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Slow loading of large-sized tiles

1 participant