Repository navigation
Description
Tracking issue for project format 2, as decided in #1378 (decision): a .rgis that links to its data with tool results beside it, and .rgispkg as the complete package. Each piece below is its own pull request; this issue keeps the order and links them. Comments on the design go here, so #1378 can rest.
The decision, in short
.rgis(format 2) is a small project file: maps, styles, layouts, the layer list and where each layer's data is. It holds no layer data. Tool results and edited copies are written as files beside it..rgispkgis the complete package: the project with all its data, for sending and archiving. Copies inside it remember their source (the snapshot and provenance model from How should RustGIS projects store their data? A proposal for the .rgis format #1378, variant 3).- Format 1 is every
.rgissaved so far. It keeps opening with its layers, as a package does, and is never rewritten on its own. Its first save asks: keep everything in one.rgispkg, or convert to a.rgiswith the data beside it.
What exists
.rgisis one JSON document (crates/engine/src/project.rs,FORMAT). A newer one is refused: "this project was saved by a newer RustGIS". Layers loaded whole and every tool result are stored in it; streamed files, rasters, databases and services are linked.- Since fix(engine): address release sweep correctness findings #2201 (v0.7.0) RustGIS writes
"format": 2with arequireslist naming the saved additions a file uses (CAPABILITIES, 15 so far). The layer data is still inside the file, as in format 1. v0.7.0 is also the first release that reads format 2, so v0.6.0 and older refuse projects saved by 0.7.0 with the "newer RustGIS" message. - Its size limit follows the computer's memory since feat(engine): the largest project follows the computer's memory (#1615) #1639 (512 MiB to 4 GiB), and fix(engine): reject projects too large to reopen before saving #1364 refuses to save what couldn't be reopened.
.rgispkg(Package Project) embeds local rasters only, base64 in the JSON, within a 512 MiB budget (128 MiB in the browser). What doesn't fit stays linked without a notice. A package opens as a new, unsaved project, and Save writes a.rgis.docs/maintaining.md"Saved files and settings": a format's version goes up when an older reader would lose something, and reading a new version ships at least one release before anything writes it. [Tracking]: Prepare compatibility safeguards for future RustGIS releases #1405 tracks the wider compatibility safeguards.
Rules for every piece
- Every format 1 project keeps opening with all its data and works as it does today. Opening never rewrites a file; the first save of a format 1 project keeps the original.
- A RustGIS that can read a new format ships a release before any RustGIS writes it. (Format 2 was first read and written in the same release, v0.7.0; see above.)
- Nothing is left out silently: a package says what it couldn't copy before it is written, and a layer whose data is missing or changed says so.
- One model behind both files: a layer means the same in a
.rgisand in a.rgispkg; only where its data lives differs.
Pieces
- Reading format 2, first. Done in feat(project): read project format 2, refusing one that requires a capability this RustGIS doesn't know (#1853 piece 1) #1975. The reader accepts
"format": 2and a list of required capabilities, and refuses a file that needs one it doesn't know, naming it. Older releases already refuse format 2 with the "newer RustGIS" message. ([Tracking]: Prepare compatibility safeguards for future RustGIS releases #1405 item 1.) fix(engine): address release sweep correctness findings #2201 then started writing format 2, with the layer data still in the file. - A schema for the project file. A versioned JSON Schema in the repository, checked against the Rust types with fixtures and a CI drift check, and a
docs/maintaining.mdrule that a PR changing what is saved updates both. A schema change alone doesn't raise the format. Part 1 is done in feat(project): a JSON Schema of the project file, generated from the saved types (#1853 piece 2, part 1) #2065 (elpidioassinaturas): the schema, fixtures,cargo xtask schema --checkin CI and the docs rule. Left: maps and layers, layouts, renderers and labels and the other parts still described as any object ($defs/Open). - Stable IDs. Layers, fields and project items keep IDs across saves, and feature IDs are kept, so links, snapshots, provenance and later merging can refer to them.
- Where a layer's data is: source references. Each layer records its source: a path relative to the project (absolute only across drives), a URL, or a named connection with its table and filter (never credentials). A missing or changed source is shown on the layer, with the last time it was checked. Read-only sources say so up front. (feat(ui): the Layers pane marks edits not published and a source file changed on disk #2058 already marks a source file changed on disk in the Layers pane.) It includes Set Data Source for any layer (Repair Data Source renamed, keeping its symbology, labels and definition query) and switching a layer between linking and copying, both ways, on the Source tab of Layer Properties (jgregoriorincon, once the source references are in). Needs 3.
- Tool results beside the project. Results are written to a folder named after the project (
Shelby.rgis→Shelby.rgis.data/), one file per layer, and linked like any other source. A result written without an output file is GeoParquet 1.1, with a bbox column on layers larger than one row group (a vector), or a Cloud-Optimized GeoTIFF (a raster): both small, read in place and copied into a package unchanged. A named output keeps its own format ("output": "roads.gpkg"), and the project links to it the same way. Each records its provenance: tool (or plugin id and version), parameters, inputs and their versions, so a result whose input changed is marked "possibly out of date", never recomputed silently. A per-layer scratch flag keeps intermediate layers out of the folder. Supersedes A project's data folder: downloaded and derived layers kept as files beside the.rgis#1653. Needs 4. - Linked data in the project file. The format number 2 is already written for projects that hold their data (fix(engine): address release sweep correctness findings #2201), so the linked layout becomes a capability within format 2 (proposal:
linked-data). A reader without it refuses by name instead of opening a project without its layers. New projects save this way: the project file plus the results folder, written atomically (a temporary file replaced in one step). A project changed on disk since it was opened isn't overwritten: Save asks. Needs 1 (released), 4, 5 and 12. - The package (Mavrick-F).
.rgispkgbecomes a zip:project.json, GeoParquet 1.1 layers and COG rasters as stored (uncompressed) members, read in place without loading the package whole, and a manifest with each member's checksum, its source reference, when it was copied and its licence and credit. Before writing, the package lists what can't be copied (a WMS, a database view without export rights) and asks. Untrusted packages are read under a strict zip profile. Existing base64 packages keep opening. A CRS block per layer (PROJJSON, compound allowed, epoch, Z units) comes here rather than in piece 11, as terrio-crs needs it. terrio-crs now does RustGIS's transforms (Transform coordinates with terrio-crs (#2032 A1) #2041); WKT and GeoTIFF keys move onto it in Read and write WKT and GeoTIFF keys through terrio-crs (#2032 A2b) #2322 (open), which adds the first coordinate-system capability,crs-wkt, for a custom system only its WKT describes. Needs 4. - Snapshots: copies that remember their source. Import copies on purpose; Add links. A copy records its source, the extent or filter and the source's version, and can be refreshed per layer. Refresh never runs on its own: on open, sources are checked in the background and a layer shows changed, unchanged, couldn't check or unreachable. Includes a window of a remote COG as a snapshot kind. Copying database data warns that the copy can be read without the database's permissions. Needs 4 and 7.
- Projects with their data inside (format 1, and format 2 without
linked-data). Such a project opens as today. Its first save asks: keep it as one.rgispkg, or convert to a.rgiswith its stored layers and results written beside it (the original kept). Recovery and autosave of format 1 projects work as now. Needs 6 and 7. - The browser. The web app opens and saves both formats from day one. With no file paths, a
.rgislinks to URLs and saved connections, and results live in the browser's own storage until saved or packaged. Large packages may open with a lower, stated memory ceiling at first. Needs 6 and 7. - 3D and 4D in packages. GeoParquet 2.0 layers once the readers people use take them (1.1 until then, since 2.0's native geometry breaks pyarrow up to 19 and current geopandas), so Z and M survive; the CRS block from piece 7 with compound systems, epochs and Z units, and what M means; reserved capability names for surfaces, point clouds and meshes; saving never drops Z or M silently. Needs 7.
- Edits to linked data. With no layer data in the
.rgis, edits and drawn layers go to their source: Save edits for the formats that can be written safely (GeoPackage and PostGIS today, more added), and for the rest a copy into the data folder, offered and said. Unsaved edits are never dropped silently. Needed before 6. Needs 4 and 5.
Status (2026-11-17): pieces 1 and 2 part 1 are in main, and v0.7.0 reads and writes format 2 (data still inside). The rest of piece 2 and piece 3 can start at once; pieces 4 to 12 have no pull request yet.
Open questions
None of these blocks pieces 1 to 6: the first two are needed before piece 7, and the last two can wait.
- Saving an opened package. Save writes a package back, or converts it to a
.rgiswith the data beside it? Proposal: Save keeps it a package; Save As offers the.rgis. - A complete package. Is "complete" a promise (refuse to package what can't be copied) or a snapshot with stated limits (list the links that remain)? Proposal: the second, listed before writing.
- The project log. Save who changed what, for merging later? It doesn't block the format if the capability list and stable IDs are in place, so it can become its own member later.
- Packages on S3. A zip whose changed members are appended, or a folder layout for cloud editing and the zip for sending. Mavrick-F's append-only saves (How should RustGIS projects store their data? A proposal for the .rgis format #1378) may answer it with piece 7.
- Map packages. Fold Package Map (
.rgismpk) into the package as a "this map only" choice, with simplifying as an option, keeping.rgismpkreadable (proposed by Mavrick-F; giswqs's call).
@opengeos/rustgis-maintainers, this turns the decision in #1378 into pieces. Please check the order and the rules before piece 1 starts; comments, and who'd like which piece, are welcome. Piece 1 is done; piece 2: elpidioassinaturas; pieces 3 to 6 and 12 are open to take while I work on terrio-crs (#2032). Piece 7: Mavrick-F; Set Data Source in piece 4: jgregoriorincon.
Activity
Mavrick-F commentedon Nov 12, 2026
Thanks for writing this up, the order looks right to me. Three things before piece 1 starts:
.rgis#1653 is superseded by piece 5, so it can be closed into this issue..rgis, and only GeoPackage and PostGIS have Save edits. If the.rgisholds no layer data, where do unsaved edits and drawn layers go? I'd extend Save edits to formats that can be written safely and offer a copy into the data folder for the rest, but it needs a piece either way.I'd like to take piece 7. The append-only saves from my prototype in #1378 fit it, and they also answer the open question about appending to packages on S3.
Mavrick-F commentedon Nov 12, 2026
Oh, one more thing, inspired by #1811
.rgismpk): Import already takes maps from a.rgispkg, so a map package is a package with one map in it. I'd fold Package Map into the package as a "this map only" choice (with simplifying as an option replacing "mobile") so we don't carry two package formats going forward. Of course we'd keep reading .rgismpk for the foreseeable future.jgregoriorincon commentedon Nov 13, 2026
Two things from the layer side, now that the order is set.
1. Switching a layer between link and copy, after it's added. Piece 8 says how the choice is made when a layer comes in (Import copies, Add links), but not how to change it later. Today a layer loaded whole can't change its source at all:
layer.repairSourceandlayer.reloadDatarefuse it on purpose (crates/engine/src/cmd/repair.rs:53,:65). I'd add to piece 8, or to piece 4:2. A question about format 1 in the meantime. Format 1 already saves linked file layers: a
SourceLayerwith itspath, opened again when the project loads.layer.addDataalready links a GeoJSON, Shapefile, GeoPackage, GeoParquet or FlatGeobuf with"stream": trueon the desktop; the interface just never offers it, and decides by size. Offering "Link to the file" in Add Data would let people keep a project small before format 2 lands, without changing the format: every release that reads format 1 already opens those links. CSV, KML and GPX would still be copied, since they have no reader that works in place, and edits would follow today's rules (the open point @Mavrick-F raised).Is that worth doing now, or would you rather leave format 1 as it is until piece 9? I'm happy to take it, or the Source tab part of point 1, whichever fits the plan.
Mavrick-F commentedon Nov 13, 2026
So basically allowing/encouraging linked data of all sizes prior to the format change? That makes sense to me.
pick2510 commentedon Nov 13, 2026
Thanks both.
@Mavrick-F:
.rgis#1653 can be closed into this issue..rgismpkinto the package as a "this map only" choice makes sense to me, with reading.rgismpkkept. @giswqs, your call.@jgregoriorincon:
pick2510 commentedon Nov 13, 2026
I am working on terrio-crs integration currently. So if you want to start with this plan, please do so
pick2510 commentedon Nov 13, 2026
I'll take piece 1 (reading format 2 and the capability list) now, beside terrio-crs.
fix(project): task, presentation, notebook and model documents in tas…
feat(project): read project format 2, refusing one that requires a ca…
elpidioassinaturas commentedon Nov 14, 2026
I'll take piece 2 (the schema for the project file).
Plan, in small PRs:
schemarsbehind an off-by-defaultschemafeature (cfg_attrderives), so the app, the browser build and the package don't compile it.schemas/project.schema.json(JSON Schema 2020-12, versioned by the project format).cargo xtask schema --checkfails CI when the types and the committed file disagree.docs/maintaining.mdrule: a PR that changes what is saved regenerates the schema in the same PR.Dataset. Maps, layers, layouts and the rest are left as open objects there, and are described crate by crate in the following PRs.pick2510 commentedon Nov 14, 2026
The coordinate system work is tracked in #2032. RustGIS now transforms through terrio-crs (#2041), and piece 7's per-layer coordinate system block can be written with terrio-crs's PROJJSON (C2 in #2032).
elpidioassinaturas commentedon Nov 14, 2026
Piece 2, part 1 is in #2065 (still a draft). Before it goes ready I'd like a maintainer's ok on one new dependency, since
CONTRIBUTING.mdasks to coordinate first:schemars1.2 (MIT; its dependenciesschemars_derive,serde_derive_internals,dyn-clone,ref-castandref-cast-implare MIT or Apache-2.0, all allowed bydeny.toml). It generates the schema from the saved types and follows serde's own attributes, so the schema can't drift from what is saved.schemafeature inrustgis-geom,rustgis-dataandrustgis-engine, used by one example and bycargo xtask schema. The desktop app, the browser (wasm) build, the CLI and the Python package never compile it, andTHIRD-PARTY-LICENSES.txtdoesn't change.Is that fine with you?
elpidioassinaturas commentedon Nov 14, 2026
@giswqs @Mavrick-F, a short ping on the
schemarsquestion above, in case #2065 can make it into tonight's release: it's the only new dependency, behind an off-by-defaultschemafeature, and the app, wasm, CLI and Python package never compile it. If you'd rather not add it, I'll hand-write the schema instead.pick2510 commentedon Nov 14, 2026
Ok for me.
Mavrick-F commentedon Nov 14, 2026
Fine with me, but it doesn't touch anything user facing, so don't rush it just to beat the next release.
feat(project): a JSON Schema of the project file, generated from the …
pick2510 commentedon Nov 17, 2026
@elpidioassinaturas, a short question on piece 2: are you planning the next parts of the schema (maps and their layers, layouts and the rest that is still
$defs/Open)? Piece 3 (stable IDs for layers) needs the maps and layers described, so it would help to know. If you'd rather hand it over, that's fine too, just say so here.