Skip to content

Tracking: large data (layers over 2 million features, slow hillshade and GeoTIFF writing) #1111

Open

Description

@Mavrick-F

On real-sized data RustGIS either refuses or is much slower than it needs to be. Measured on one US state (3,212,306 building footprints, 4,562 census block groups, a 9667 × 11947 DEM; rustgis-cli at d58cb8a, i7-6700, single runs, ±15%), with the time an established tool takes on the same files in brackets. This issue tracks fixing that, and has one decision to make first.

What's wrong now

Needs a decision: tools on layers bigger than memory

Steps 2 and 3 below happen either way: exports stream, and the count becomes a memory budget. What should tools do past the budget?

  • A. Streaming tools (large). Per-feature tools (reproject, buffer, calculate geometry) work a chunk at a time. Tools that compare two layers (count points, spatial join, clip, select by location) index the smaller layer and stream the larger. Outputs go to temporary files added as streamed layers. That means tools on any size, but every tool gets a second execution path, and temporary-file outputs touch undo, project saving and the web app, which has no temporary files.
  • B. Stay in memory (small to very large). Tools keep working on a whole layer under the budget, and past it the answer is "clip or filter first". There is one execution path, but a ceiling, lower in the web app. Small changes (boxing the common one-ring polygon, reserving from the known count) might take a feature from about 355 to 250 bytes. A flat, columnar geometry model would roughly halve memory, but touches every crate.

Which fits where RustGIS is heading?

The steps also assume these defaults. Say so if any is wrong:

  • hillshade writes Byte;
  • threads on the desktop only, with the web app single-threaded as the renderer already is (threads_for);
  • a budget of 4 GiB on the desktop and 1 GiB in the web app, adjustable;
  • COGs use the floating-point predictor (TIFF Predictor 3), which GDAL, QGIS and the tiff crate read.

Plan

1. Correctness (small, independent)

2. Stream exports and reprojection (medium)

3. A memory budget instead of MAX_LIMIT (small, after 2)

  • load_all counts bytes as it reads chunks and refuses past the budget, with try_reserve for the large vectors. Done in perf(data): read a whole layer within a memory budget instead of a feature count #1255: GeoPackage, FlatGeobuf, GeoParquet and a Shapefile read in place have no count limit, up to 4 GiB of features (1 GiB in the web app); all 3.2M buildings load in 5.9 s, 1.22 GB. The budget is a constant, not yet adjustable. Other sources keep the 2,000,000 count.

4. Terrain kernels (small, independent)

5. GeoTIFF writing (small to medium, independent)

6. Threads on the desktop (medium, after 4 and 5)

7. Tools past the budget (large or skipped, after the decision)

  • A: streaming tools, Summarize Within first. B: the small memory savings.

Related

#1105 and #1108 are filed separately because they're bugs whatever is decided here. Step 2 builds on #681's chunked reading. #1065 (layers read in place are read twice on every render) is about drawing, not tools, so it stays separate.

Activity

  1. choupeanut commented on Oct 29, 2026

    @choupeanut
    Contributor

    I will take a small, independent part of step 4: hoist the invariant cell-size calculation for slope, aspect and hillshade, and the invariant light angles for hillshade, while retaining the existing formulas, output type and no-data behavior. I will compare results against the current implementation and measure the speed on a representative raster. I see #1085 separately corrects axis orientation; this change will not alter that behavior or take over its fix.

  2. choupeanut commented on Oct 29, 2026

    @choupeanut
    Contributor

    I will take the remaining terrain row-pass item in step 4, separate from the merged constant-hoisting change. I will retain the existing edge, no-data and axis-orientation behavior, compare slope, aspect and hillshade against the current implementation, and measure the kernels before and after. This adds no dependency or output-format change.

  3. choupeanut commented on Oct 29, 2026

    @choupeanut
    Contributor

    I will take the GeoPackage portion of step 2: implement complete, ordered, byte-bounded visit_chunks traversal for the native and in-memory GeoPackage readers, with extent, invalid-row and cancellation coverage. This is the reader groundwork only; streaming writers and export wiring remain separate. No dependency, format or command changes.

  4. Mavrick-F commented on Oct 29, 2026

    @Mavrick-F
    MemberAuthor

    I'm taking step 2 (stream exports and reprojection), as a stack of three pull requests:

    1. visit_chunks for GeoPackage, FlatGeobuf and GeoParquet, behind a new visits_chunks() capability. streams_chunks() also makes the map draw a source by walking all of it on every render ([Feature]: Layers read in place are read and projected in full twice on every render #1065), so it stays off for these formats and drawing is unchanged.
    2. A writer per format that takes chunks, and share.exportLayer on a streamed layer reading chunk → reproject → writer, so exports no longer hold the whole layer or stop at MAX_LIMIT.
    3. FlatGeobuf written unindexed in the web app.

    The first is up now; it adds no dependency and changes no command or file format.

  5. choupeanut commented on Oct 29, 2026

    @choupeanut
    Contributor

    I see #1131 now covers the GeoPackage traversal as well as the other step 2 readers, with the separate traversal capability and an independent SQLite connection. I am stepping back from my overlapping GeoPackage work so there is no duplicate PR. Thanks!

  6. Mavrick-F commented on Oct 29, 2026

    @Mavrick-F
    MemberAuthor

    @choupeanut Thanks, sorry for missing your note about starting the GeoPackage work!

  7. Mavrick-F commented on Oct 29, 2026

    @Mavrick-F
    MemberAuthor

    Starting the second pull request of step 2 now, on top of #1131: share.exportLayer on a streamed layer reads it a chunk at a time, reprojects each chunk in place and writes it straight to the file, for GeoJSON, CSV, FlatGeobuf, GeoPackage and GeoParquet on the desktop. Those formats get one writer each, used by both the in-memory and the streamed export, so the two give the same file. Exports in these formats then no longer stop at MAX_LIMIT.

    Shapefile, KML, KMZ and GPX keep the whole-layer path for now: a Shapefile's DBF field widths come from a pass over the data, so streaming it needs two passes, which I'd rather do as its own change. The web app's export is step 2's third item.

  8. Mavrick-F commented on Oct 29, 2026

    @Mavrick-F
    MemberAuthor

    Two updates from a full run of the measurements behind this issue (four sizes, from 19,000 to 3.2 million buildings, upstream main at 13aee13 plus #1119):

  9. Mavrick-F commented on Nov 1, 2026

    @Mavrick-F
    MemberAuthor

    I'm taking step 3 in #1255: a whole read of a GeoPackage, FlatGeobuf or GeoParquet file, or a Shapefile read in place, now counts the memory its features hold and stops at 4 GiB (1 GiB in the web app) instead of 2,000,000 features. All 3.2 million buildings now load in 5.9 s with a 1.22 GB peak. The budget is a constant for now, not yet adjustable. Other sources keep the count limit. No dependency, command or format change.

  10. Mavrick-F commented on Nov 12, 2026

    @Mavrick-F
    MemberAuthor

    I'm taking the first item of step 5: the COG writer compresses its tiles with Deflate level 1 and a predictor (the floating-point one for float samples), so the COG export and Project Raster with a path spend less time writing and write smaller files. I'll measure the 9667 × 11947 DEM before and after. No dependency, command or file-format change; the untiled export (the second item) stays separate.

  11. Mavrick-F commented on Nov 12, 2026

    @Mavrick-F
    MemberAuthor

    I'm taking the second item of step 4: a hillshade from Raster Functions (and a function chain ending in one) is made as bytes, its shades 1–255 with 0 for no data, so a cell in full shadow is 1 instead of 0. The untiled GeoTIFF writer then writes an 8-bit grey band as bytes, as the COG writer already does, so the export and the copy a project keeps are a quarter of the size. The layer is still drawn as a grey stretch. No dependency, command or file-format change.

  12. Mavrick-F commented on Nov 13, 2026

    @Mavrick-F
    MemberAuthor

    I'm taking the second item of step 5: the untiled GeoTIFF goes through the in-house writer the COG export uses, so its strips get the floating-point predictor (and differencing for bytes) at Deflate's fastest level, with classes kept at the default level without a predictor, as in #1860. The samples and tags stay as they are. No dependency, command or format change.

  13. Mavrick-F commented on Nov 13, 2026

    @Mavrick-F
    MemberAuthor

    I'm taking the COG half of step 6: the COG writer's tiles, and the untiled GeoTIFF's strips, are made on several threads on the desktop while one thread writes them in their order, so the file stays the same byte for byte. On the DEM the COG export takes about half the time. No dependency, command or format change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions