Repository navigation
Description
On real-sized data RustGIS either refuses or is much slower than it needs to be. Measured on one US state (3,212,306 building footprints, 4,562 census block groups, a 9667 × 11947 DEM; rustgis-cli at d58cb8a, i7-6700, single runs, ±15%), with the time an established tool takes on the same files in brackets. This issue tracks fixing that, and has one decision to make first.
What's wrong now
- Layers over 2,000,000 features can't be exported, reprojected or used in tools.
MAX_LIMIT(crates/data/src/source/mod.rs) refuses them: all 3.2M buildings can't be converted, and [Bug]: A source over the 2,000,000-feature limit is silently replaced by a sample when read without streaming: exports and tools run on half the data #1108 is worse, a silent half sample. It's a count, not memory: it refuses 3.2M small polygons that fit in about 1.2 GB, but doesn't stop 2M large ones from exhausting memory. - Exports hold the whole layer, sometimes twice. Converting 1.79M polygons to FlatGeobuf takes 22.7 s and 1.2 GB (8.3 s, 0.26 GB). Reprojection clones the
Dataset, and the FlatGeobuf writer builds the whole file in aVec<u8>. - Hillshade is slow. It takes 59 s (7 s).
gradient()incrates/analysis/src/rasterops.rslooks up the coordinate system for every cell, 115M times here, and runs five trig functions per cell. Its output is written as Float32 although the values are whole numbers 0–255. - Writing GeoTIFFs is slow. The COG export takes 43 s (16–22 s), nearly all of it Deflate at level 6 through
miniz_oxide, about 18 MB/s (crates/data/src/raster/cogwrite.rs). - Summarize Within compares every point with every polygon ([Bug]: Summarize Within (countPoints) tests every point against every polygon: 1.8 million points in 4,562 polygons run for over 28 minutes #1105): over 28 minutes for 1.79M points (15 s).
Needs a decision: tools on layers bigger than memory
Steps 2 and 3 below happen either way: exports stream, and the count becomes a memory budget. What should tools do past the budget?
- A. Streaming tools (large). Per-feature tools (reproject, buffer, calculate geometry) work a chunk at a time. Tools that compare two layers (count points, spatial join, clip, select by location) index the smaller layer and stream the larger. Outputs go to temporary files added as streamed layers. That means tools on any size, but every tool gets a second execution path, and temporary-file outputs touch undo, project saving and the web app, which has no temporary files.
- B. Stay in memory (small to very large). Tools keep working on a whole layer under the budget, and past it the answer is "clip or filter first". There is one execution path, but a ceiling, lower in the web app. Small changes (boxing the common one-ring polygon, reserving from the known count) might take a feature from about 355 to 250 bytes. A flat, columnar geometry model would roughly halve memory, but touches every crate.
Which fits where RustGIS is heading?
The steps also assume these defaults. Say so if any is wrong:
- hillshade writes Byte;
- threads on the desktop only, with the web app single-threaded as the renderer already is (
threads_for); - a budget of 4 GiB on the desktop and 1 GiB in the web app, adjustable;
- COGs use the floating-point predictor (TIFF Predictor 3), which GDAL, QGIS and the
tiffcrate read.
Plan
1. Correctness (small, independent)
- Refuse instead of sampling ([Bug]: A source over the 2,000,000-feature limit is silently replaced by a sample when read without streaming: exports and tools run on half the data #1108): an incomplete read in
source::materializebecomes an error, checked from the known count before reading. Done in fix(data): refuse a source over the read limit instead of reading a sample as the whole of it #1116. - Index Summarize Within ([Bug]: Summarize Within (countPoints) tests every point against every polygon: 1.8 million points in 4,562 polygons run for over 28 minutes #1105): the polygons in
ExtentIndex, each point tested against its candidates only. Done in perf(analysis): index the polygons in countPoints #1119; Clip and Select by Location followed in perf(analysis): index Clip and Select by Location candidates #1148 ([Bug]: Clip and Select by Location compare every feature with every overlay polygon: minutes where an indexed tool takes seconds #1142).
2. Stream exports and reprojection (medium)
-
visit_chunksfor GeoPackage, FlatGeobuf and GeoParquet. Shapefile already has it (feat(render): stream complete Shapefile layers in ordered chunks #681). Done in feat(data): read GeoPackage, FlatGeobuf and GeoParquet a chunk at a time #1131, behind a separatevisits_chunks()capability so drawing is unchanged. - A writer per format that takes chunks, and
share.exportLayeron a streamed layer reading chunk → reproject → writer. Memory goes from 1.2 GB to about 0.3 GB, and the limit no longer applies to exports. Done in perf(data): export a streamed layer a chunk at a time #1146 for GeoJSON, CSV, FlatGeobuf, GeoPackage and GeoParquet (3.2M polygons to GeoPackage in 0.20 GB). Shapefile, KML, KMZ and GPX still export the whole layer: a Shapefile's field widths need a pass over the data first. - FlatGeobuf in the web app: its index needs a temporary file, so write it unindexed there. Blocked: the
flatgeobufcrate's writer creates a temporary file even without an index. Until [rust] Hold features in memory on wasm32-unknown-unknown flatgeobuf/flatgeobuf#543 (an unindexed writer with no temporary file) is released, the web app refuses FlatGeobuf with a message instead of aborting (fix(data): refuse FlatGeobuf in the web app instead of aborting it #1152).
3. A memory budget instead of MAX_LIMIT (small, after 2)
-
load_allcounts bytes as it reads chunks and refuses past the budget, withtry_reservefor the large vectors. Done in perf(data): read a whole layer within a memory budget instead of a feature count #1255: GeoPackage, FlatGeobuf, GeoParquet and a Shapefile read in place have no count limit, up to 4 GiB of features (1 GiB in the web app); all 3.2M buildings load in 5.9 s, 1.22 GB. The budget is a constant, not yet adjustable. Other sources keep the 2,000,000 count.
4. Terrain kernels (small, independent)
- Hillshade, slope and aspect as row passes: constants computed once, three row slices walked with
windows(3)andzip, with no-data and edges on the existing path. A standalone test gave identical output on all 115M cells, 21 s → 4 s on one thread, in the web app too. Done in perf(analysis): reuse terrain raster constants #1122 (constants once) and perf(analysis): process terrain gradients by row #1130 (row windows); Tracking: performance gaps from Geoawesome's QGIS benchmark (buffer, raster, drawing) #1227 asks for a re-measure against the benchmark. - Hillshade writes Byte. Done in perf(analysis): hillshade is made and written as bytes (#1111) #1887: shades 1–255 with 0 for no data, drawn as before; the untiled GeoTIFF writer now writes any 8-bit grey band as bytes, as the COG writer did.
5. GeoTIFF writing (small to medium, independent)
- Deflate level 1 with Predictor 3 in
cogwrite.rs. Done in perf(data): a COG's tiles compressed after a predictor, at Deflate's fastest level (#1111) #1860: floats take Predictor 3, 8- and 16-bit whole numbers Predictor 2, classes keep level 6 with none. On the DEM the COG export takes 21 s of CPU time instead of 41–58 s, Project Raster with apath45 s instead of about 80 s, and both files are 22% smaller. - The untiled export through the in-house writer. Done in perf(data): the untiled GeoTIFF export takes a predictor and Deflate's fastest level (#1111) #1886: its strips take the same predictors and Deflate level as the COG's tiles. On the DEM, writing takes about 7 s of CPU time instead of about 20, and the file is 25% smaller (376 → 282 MB).
6. Threads on the desktop (medium, after 4 and 5)
- Raster kernels in row bands, COG tiles compressed in parallel: about 3× on 4 cores. The raster kernels are now tracked with the other tools in Tracking: analysis tools on several cores #1252, on its shared helper; the COG tiles stay here. Done in perf(data): compress GeoTIFF tiles and strips on several threads (#1111) #1921: the COG's tiles and the untiled export's strips are made on several threads and written in order, the same file byte for byte. On the DEM the COG export takes 6.6–11.7 s instead of 15.1–23.4 s, the untiled export 4.8–9.9 s instead of 7.7–12.1 s.
7. Tools past the budget (large or skipped, after the decision)
- A: streaming tools, Summarize Within first. B: the small memory savings.
Related
#1105 and #1108 are filed separately because they're bugs whatever is decided here. Step 2 builds on #681's chunked reading. #1065 (layers read in place are read twice on every render) is about drawing, not tools, so it stays separate.
Activity
choupeanut commentedon Oct 29, 2026
I will take a small, independent part of step 4: hoist the invariant cell-size calculation for slope, aspect and hillshade, and the invariant light angles for hillshade, while retaining the existing formulas, output type and no-data behavior. I will compare results against the current implementation and measure the speed on a representative raster. I see #1085 separately corrects axis orientation; this change will not alter that behavior or take over its fix.
choupeanut commentedon Oct 29, 2026
I will take the remaining terrain row-pass item in step 4, separate from the merged constant-hoisting change. I will retain the existing edge, no-data and axis-orientation behavior, compare slope, aspect and hillshade against the current implementation, and measure the kernels before and after. This adds no dependency or output-format change.
choupeanut commentedon Oct 29, 2026
I will take the GeoPackage portion of step 2: implement complete, ordered, byte-bounded visit_chunks traversal for the native and in-memory GeoPackage readers, with extent, invalid-row and cancellation coverage. This is the reader groundwork only; streaming writers and export wiring remain separate. No dependency, format or command changes.
Mavrick-F commentedon Oct 29, 2026
I'm taking step 2 (stream exports and reprojection), as a stack of three pull requests:
visit_chunksfor GeoPackage, FlatGeobuf and GeoParquet, behind a newvisits_chunks()capability.streams_chunks()also makes the map draw a source by walking all of it on every render ([Feature]: Layers read in place are read and projected in full twice on every render #1065), so it stays off for these formats and drawing is unchanged.share.exportLayeron a streamed layer reading chunk → reproject → writer, so exports no longer hold the whole layer or stop atMAX_LIMIT.The first is up now; it adds no dependency and changes no command or file format.
choupeanut commentedon Oct 29, 2026
I see #1131 now covers the GeoPackage traversal as well as the other step 2 readers, with the separate traversal capability and an independent SQLite connection. I am stepping back from my overlapping GeoPackage work so there is no duplicate PR. Thanks!
Mavrick-F commentedon Oct 29, 2026
@choupeanut Thanks, sorry for missing your note about starting the GeoPackage work!
Mavrick-F commentedon Oct 29, 2026
Starting the second pull request of step 2 now, on top of #1131:
share.exportLayeron a streamed layer reads it a chunk at a time, reprojects each chunk in place and writes it straight to the file, for GeoJSON, CSV, FlatGeobuf, GeoPackage and GeoParquet on the desktop. Those formats get one writer each, used by both the in-memory and the streamed export, so the two give the same file. Exports in these formats then no longer stop atMAX_LIMIT.Shapefile, KML, KMZ and GPX keep the whole-layer path for now: a Shapefile's DBF field widths come from a pass over the data, so streaming it needs two passes, which I'd rather do as its own change. The web app's export is step 2's third item.
Mavrick-F commentedon Oct 29, 2026
Two updates from a full run of the measurements behind this issue (four sizes, from 19,000 to 3.2 million buildings, upstream main at 13aee13 plus #1119):
path,raster.projectwrites its output throughcogwrite, so it pays the same Deflate cost: projecting the 9667 × 11947 DEM took 48 s, of which writing is about 35 s (the COG export of the same DEM alone took 35 s). The warp itself is about 13 s, on par withgdalwarp(15 s including its own write).feat(data): read GeoPackage, FlatGeobuf and GeoParquet a chunk at a t…
perf(data): export a streamed layer a chunk at a time (#1146)
fix(data): refuse FlatGeobuf in the web app instead of aborting it (#…
Mavrick-F commentedon Nov 1, 2026
I'm taking step 3 in #1255: a whole read of a GeoPackage, FlatGeobuf or GeoParquet file, or a Shapefile read in place, now counts the memory its features hold and stops at 4 GiB (1 GiB in the web app) instead of 2,000,000 features. All 3.2 million buildings now load in 5.9 s with a 1.22 GB peak. The budget is a constant for now, not yet adjustable. Other sources keep the count limit. No dependency, command or format change.
perf(data): read a whole layer within a memory budget instead of a fe…
perf(analysis): slope, aspect and hillshade rows on several cores (#1252
Mavrick-F commentedon Nov 12, 2026
I'm taking the first item of step 5: the COG writer compresses its tiles with Deflate level 1 and a predictor (the floating-point one for float samples), so the COG export and Project Raster with a
pathspend less time writing and write smaller files. I'll measure the 9667 × 11947 DEM before and after. No dependency, command or file-format change; the untiled export (the second item) stays separate.perf(data): a COG's tiles compressed after a predictor, at Deflate's …
Mavrick-F commentedon Nov 12, 2026
I'm taking the second item of step 4: a hillshade from Raster Functions (and a function chain ending in one) is made as bytes, its shades 1–255 with 0 for no data, so a cell in full shadow is 1 instead of 0. The untiled GeoTIFF writer then writes an 8-bit grey band as bytes, as the COG writer already does, so the export and the copy a project keeps are a quarter of the size. The layer is still drawn as a grey stretch. No dependency, command or file-format change.
Mavrick-F commentedon Nov 13, 2026
I'm taking the second item of step 5: the untiled GeoTIFF goes through the in-house writer the COG export uses, so its strips get the floating-point predictor (and differencing for bytes) at Deflate's fastest level, with classes kept at the default level without a predictor, as in #1860. The samples and tags stay as they are. No dependency, command or format change.
perf(analysis): hillshade is made and written as bytes (#1111) (#1887)
perf(data): the untiled GeoTIFF export takes a predictor and Deflate'…
Mavrick-F commentedon Nov 13, 2026
I'm taking the COG half of step 6: the COG writer's tiles, and the untiled GeoTIFF's strips, are made on several threads on the desktop while one thread writes them in their order, so the file stays the same byte for byte. On the DEM the COG export takes about half the time. No dependency, command or format change.
perf(data): compress GeoTIFF tiles and strips on several threads (#1111…