Skip to content

Tracking: project format 2, a .rgis that links to its data and .rgispkg as the complete package #1853

Open

Description

@pick2510

Tracking issue for project format 2, as decided in #1378 (decision): a .rgis that links to its data with tool results beside it, and .rgispkg as the complete package. Each piece below is its own pull request; this issue keeps the order and links them. Comments on the design go here, so #1378 can rest.

The decision, in short

  • .rgis (format 2) is a small project file: maps, styles, layouts, the layer list and where each layer's data is. It holds no layer data. Tool results and edited copies are written as files beside it.
  • .rgispkg is the complete package: the project with all its data, for sending and archiving. Copies inside it remember their source (the snapshot and provenance model from How should RustGIS projects store their data? A proposal for the .rgis format #1378, variant 3).
  • Format 1 is every .rgis saved so far. It keeps opening with its layers, as a package does, and is never rewritten on its own. Its first save asks: keep everything in one .rgispkg, or convert to a .rgis with the data beside it.

What exists

  • .rgis is one JSON document (crates/engine/src/project.rs, FORMAT). A newer one is refused: "this project was saved by a newer RustGIS". Layers loaded whole and every tool result are stored in it; streamed files, rasters, databases and services are linked.
  • Since fix(engine): address release sweep correctness findings #2201 (v0.7.0) RustGIS writes "format": 2 with a requires list naming the saved additions a file uses (CAPABILITIES, 15 so far). The layer data is still inside the file, as in format 1. v0.7.0 is also the first release that reads format 2, so v0.6.0 and older refuse projects saved by 0.7.0 with the "newer RustGIS" message.
  • Its size limit follows the computer's memory since feat(engine): the largest project follows the computer's memory (#1615) #1639 (512 MiB to 4 GiB), and fix(engine): reject projects too large to reopen before saving #1364 refuses to save what couldn't be reopened.
  • .rgispkg (Package Project) embeds local rasters only, base64 in the JSON, within a 512 MiB budget (128 MiB in the browser). What doesn't fit stays linked without a notice. A package opens as a new, unsaved project, and Save writes a .rgis.
  • docs/maintaining.md "Saved files and settings": a format's version goes up when an older reader would lose something, and reading a new version ships at least one release before anything writes it. [Tracking]: Prepare compatibility safeguards for future RustGIS releases #1405 tracks the wider compatibility safeguards.

Rules for every piece

  • Every format 1 project keeps opening with all its data and works as it does today. Opening never rewrites a file; the first save of a format 1 project keeps the original.
  • A RustGIS that can read a new format ships a release before any RustGIS writes it. (Format 2 was first read and written in the same release, v0.7.0; see above.)
  • Nothing is left out silently: a package says what it couldn't copy before it is written, and a layer whose data is missing or changed says so.
  • One model behind both files: a layer means the same in a .rgis and in a .rgispkg; only where its data lives differs.

Pieces

  1. Reading format 2, first. Done in feat(project): read project format 2, refusing one that requires a capability this RustGIS doesn't know (#1853 piece 1) #1975. The reader accepts "format": 2 and a list of required capabilities, and refuses a file that needs one it doesn't know, naming it. Older releases already refuse format 2 with the "newer RustGIS" message. ([Tracking]: Prepare compatibility safeguards for future RustGIS releases #1405 item 1.) fix(engine): address release sweep correctness findings #2201 then started writing format 2, with the layer data still in the file.
  2. A schema for the project file. A versioned JSON Schema in the repository, checked against the Rust types with fixtures and a CI drift check, and a docs/maintaining.md rule that a PR changing what is saved updates both. A schema change alone doesn't raise the format. Part 1 is done in feat(project): a JSON Schema of the project file, generated from the saved types (#1853 piece 2, part 1) #2065 (elpidioassinaturas): the schema, fixtures, cargo xtask schema --check in CI and the docs rule. Left: maps and layers, layouts, renderers and labels and the other parts still described as any object ($defs/Open).
  3. Stable IDs. Layers, fields and project items keep IDs across saves, and feature IDs are kept, so links, snapshots, provenance and later merging can refer to them.
  4. Where a layer's data is: source references. Each layer records its source: a path relative to the project (absolute only across drives), a URL, or a named connection with its table and filter (never credentials). A missing or changed source is shown on the layer, with the last time it was checked. Read-only sources say so up front. (feat(ui): the Layers pane marks edits not published and a source file changed on disk #2058 already marks a source file changed on disk in the Layers pane.) It includes Set Data Source for any layer (Repair Data Source renamed, keeping its symbology, labels and definition query) and switching a layer between linking and copying, both ways, on the Source tab of Layer Properties (jgregoriorincon, once the source references are in). Needs 3.
  5. Tool results beside the project. Results are written to a folder named after the project (Shelby.rgis → Shelby.rgis.data/), one file per layer, and linked like any other source. A result written without an output file is GeoParquet 1.1, with a bbox column on layers larger than one row group (a vector), or a Cloud-Optimized GeoTIFF (a raster): both small, read in place and copied into a package unchanged. A named output keeps its own format ("output": "roads.gpkg"), and the project links to it the same way. Each records its provenance: tool (or plugin id and version), parameters, inputs and their versions, so a result whose input changed is marked "possibly out of date", never recomputed silently. A per-layer scratch flag keeps intermediate layers out of the folder. Supersedes A project's data folder: downloaded and derived layers kept as files beside the .rgis #1653. Needs 4.
  6. Linked data in the project file. The format number 2 is already written for projects that hold their data (fix(engine): address release sweep correctness findings #2201), so the linked layout becomes a capability within format 2 (proposal: linked-data). A reader without it refuses by name instead of opening a project without its layers. New projects save this way: the project file plus the results folder, written atomically (a temporary file replaced in one step). A project changed on disk since it was opened isn't overwritten: Save asks. Needs 1 (released), 4, 5 and 12.
  7. The package (Mavrick-F). .rgispkg becomes a zip: project.json, GeoParquet 1.1 layers and COG rasters as stored (uncompressed) members, read in place without loading the package whole, and a manifest with each member's checksum, its source reference, when it was copied and its licence and credit. Before writing, the package lists what can't be copied (a WMS, a database view without export rights) and asks. Untrusted packages are read under a strict zip profile. Existing base64 packages keep opening. A CRS block per layer (PROJJSON, compound allowed, epoch, Z units) comes here rather than in piece 11, as terrio-crs needs it. terrio-crs now does RustGIS's transforms (Transform coordinates with terrio-crs (#2032 A1) #2041); WKT and GeoTIFF keys move onto it in Read and write WKT and GeoTIFF keys through terrio-crs (#2032 A2b) #2322 (open), which adds the first coordinate-system capability, crs-wkt, for a custom system only its WKT describes. Needs 4.
  8. Snapshots: copies that remember their source. Import copies on purpose; Add links. A copy records its source, the extent or filter and the source's version, and can be refreshed per layer. Refresh never runs on its own: on open, sources are checked in the background and a layer shows changed, unchanged, couldn't check or unreachable. Includes a window of a remote COG as a snapshot kind. Copying database data warns that the copy can be read without the database's permissions. Needs 4 and 7.
  9. Projects with their data inside (format 1, and format 2 without linked-data). Such a project opens as today. Its first save asks: keep it as one .rgispkg, or convert to a .rgis with its stored layers and results written beside it (the original kept). Recovery and autosave of format 1 projects work as now. Needs 6 and 7.
  10. The browser. The web app opens and saves both formats from day one. With no file paths, a .rgis links to URLs and saved connections, and results live in the browser's own storage until saved or packaged. Large packages may open with a lower, stated memory ceiling at first. Needs 6 and 7.
  11. 3D and 4D in packages. GeoParquet 2.0 layers once the readers people use take them (1.1 until then, since 2.0's native geometry breaks pyarrow up to 19 and current geopandas), so Z and M survive; the CRS block from piece 7 with compound systems, epochs and Z units, and what M means; reserved capability names for surfaces, point clouds and meshes; saving never drops Z or M silently. Needs 7.
  12. Edits to linked data. With no layer data in the .rgis, edits and drawn layers go to their source: Save edits for the formats that can be written safely (GeoPackage and PostGIS today, more added), and for the rest a copy into the data folder, offered and said. Unsaved edits are never dropped silently. Needed before 6. Needs 4 and 5.

Status (2026-11-17): pieces 1 and 2 part 1 are in main, and v0.7.0 reads and writes format 2 (data still inside). The rest of piece 2 and piece 3 can start at once; pieces 4 to 12 have no pull request yet.

Open questions

None of these blocks pieces 1 to 6: the first two are needed before piece 7, and the last two can wait.

  • Saving an opened package. Save writes a package back, or converts it to a .rgis with the data beside it? Proposal: Save keeps it a package; Save As offers the .rgis.
  • A complete package. Is "complete" a promise (refuse to package what can't be copied) or a snapshot with stated limits (list the links that remain)? Proposal: the second, listed before writing.
  • The project log. Save who changed what, for merging later? It doesn't block the format if the capability list and stable IDs are in place, so it can become its own member later.
  • Packages on S3. A zip whose changed members are appended, or a folder layout for cloud editing and the zip for sending. Mavrick-F's append-only saves (How should RustGIS projects store their data? A proposal for the .rgis format #1378) may answer it with piece 7.
  • Map packages. Fold Package Map (.rgismpk) into the package as a "this map only" choice, with simplifying as an option, keeping .rgismpk readable (proposed by Mavrick-F; giswqs's call).

@opengeos/rustgis-maintainers, this turns the decision in #1378 into pieces. Please check the order and the rules before piece 1 starts; comments, and who'd like which piece, are welcome. Piece 1 is done; piece 2: elpidioassinaturas; pieces 3 to 6 and 12 are open to take while I work on terrio-crs (#2032). Piece 7: Mavrick-F; Set Data Source in piece 4: jgregoriorincon.

Activity

  1. pinned this issue on Nov 12, 2026
  2. Mavrick-F commented on Nov 12, 2026

    @Mavrick-F
    Member

    Thanks for writing this up, the order looks right to me. Three things before piece 1 starts:

    • A project's data folder: downloaded and derived layers kept as files beside the .rgis #1653 is superseded by piece 5, so it can be closed into this issue.
    • Edits to linked data have no piece yet. Today, Shapefile and GeoJSON edits are saved inside the .rgis, and only GeoPackage and PostGIS have Save edits. If the .rgis holds no layer data, where do unsaved edits and drawn layers go? I'd extend Save edits to formats that can be written safely and offer a copy into the data folder for the rest, but it needs a piece either way.
    • Results and package layers should stay GeoParquet 1.1 for now. I tested GDAL's compatibility mode (native geometry type plus 1.1 metadata): anything on Arrow 19 or older, including pyarrow up to 19 and GDAL builds on it, refuses the whole file, and native-only breaks current geopandas. 1.1 with a bbox column on layers larger than one row group keeps region reads fast (it adds about 18–23% to small-feature layers, nothing to small results), and packages can still copy results unchanged. Moving to 2.0 later is just a change to what we write. The CRS block per layer could also come before piece 11, since terrio-crs needs it.

    I'd like to take piece 7. The append-only saves from my prototype in #1378 fit it, and they also answer the open question about appending to packages on S3.

  3. Mavrick-F commented on Nov 12, 2026

    @Mavrick-F
    Member

    Oh, one more thing, inspired by #1811

    • Map packages (.rgismpk): Import already takes maps from a .rgispkg, so a map package is a package with one map in it. I'd fold Package Map into the package as a "this map only" choice (with simplifying as an option replacing "mobile") so we don't carry two package formats going forward. Of course we'd keep reading .rgismpk for the foreseeable future.
  4. jgregoriorincon commented on Nov 13, 2026

    @jgregoriorincon
    Contributor

    Two things from the layer side, now that the order is set.

    1. Switching a layer between link and copy, after it's added. Piece 8 says how the choice is made when a layer comes in (Import copies, Add links), but not how to change it later. Today a layer loaded whole can't change its source at all: layer.repairSource and layer.reloadData refuse it on purpose (crates/engine/src/cmd/repair.rs:53, :65). I'd add to piece 8, or to piece 4:

    • Set Data Source for any layer (Repair Data Source renamed), keeping its symbology, labels and definition query.
    • Copy into the project and Link to the file per layer, both ways.
    • Both on the Source tab of Layer Properties ([Feature]: Layer Properties: one place for a layer's settings #1803), which already shows where a layer's data is and holds Repair, Reload and Define Projection.

    2. A question about format 1 in the meantime. Format 1 already saves linked file layers: a SourceLayer with its path, opened again when the project loads. layer.addData already links a GeoJSON, Shapefile, GeoPackage, GeoParquet or FlatGeobuf with "stream": true on the desktop; the interface just never offers it, and decides by size. Offering "Link to the file" in Add Data would let people keep a project small before format 2 lands, without changing the format: every release that reads format 1 already opens those links. CSV, KML and GPX would still be copied, since they have no reader that works in place, and edits would follow today's rules (the open point @Mavrick-F raised).

    Is that worth doing now, or would you rather leave format 1 as it is until piece 9? I'm happy to take it, or the Source tab part of point 1, whichever fits the plan.

  5. Mavrick-F commented on Nov 13, 2026

    @Mavrick-F
    Member

    So basically allowing/encouraging linked data of all sizes prior to the format change? That makes sense to me.

  6. pick2510 commented on Nov 13, 2026

    @pick2510
    ContributorAuthor

    Thanks both.

    @Mavrick-F:

    • A project's data folder: downloaded and derived layers kept as files beside the .rgis #1653 can be closed into this issue.
    • Edits to linked data: agreed, it needs a piece. I'll add one after piece 5: Save edits for the formats that can be written safely, a copy into the data folder for the rest.
    • GeoParquet 1.1 for now, with a bbox column on larger layers. I'll update pieces 5, 7 and 11. The CRS block per layer moves before piece 11, as terrio-crs needs it anyway.
    • Piece 7 is yours.
    • Folding .rgismpk into the package as a "this map only" choice makes sense to me, with reading .rgismpk kept. @giswqs, your call.

    @jgregoriorincon:

    1. Set Data Source and switching between link and copy fit piece 4, since they depend on how a layer records its source. Happy for you to take that part, starting with the Source tab, once piece 4's source references are in.
    2. "Link to the file" in Add Data now: yes, go ahead. It needs no format change and points the same way as format 2. Please say in the dialog that a linked layer is streamed, so editing and Define Projection need Load All Features first, and check that the links are saved relative to the project where possible.
  7. pick2510 commented on Nov 13, 2026

    @pick2510
    ContributorAuthor

    I am working on terrio-crs integration currently. So if you want to start with this plan, please do so

  8. pick2510 commented on Nov 13, 2026

    @pick2510
    ContributorAuthor

    I'll take piece 1 (reading format 2 and the capability list) now, beside terrio-crs.

  9. elpidioassinaturas commented on Nov 14, 2026

    @elpidioassinaturas
    Contributor

    I'll take piece 2 (the schema for the project file).

    Plan, in small PRs:

    • schemars behind an off-by-default schema feature (cfg_attr derives), so the app, the browser build and the package don't compile it.
    • The schema is generated from the Rust types into schemas/project.schema.json (JSON Schema 2020-12, versioned by the project format). cargo xtask schema --check fails CI when the types and the committed file disagree.
    • Fixtures (format 1 projects, and format 2 once something writes it) are checked against the schema.
    • A docs/maintaining.md rule: a PR that changes what is saved regenerates the schema in the same PR.
    • First PR: the project's top level, standalone tables and Dataset. Maps, layers, layouts and the rest are left as open objects there, and are described crate by crate in the following PRs.
  10. pick2510 commented on Nov 14, 2026

    @pick2510
    ContributorAuthor

    The coordinate system work is tracked in #2032. RustGIS now transforms through terrio-crs (#2041), and piece 7's per-layer coordinate system block can be written with terrio-crs's PROJJSON (C2 in #2032).

  11. elpidioassinaturas commented on Nov 14, 2026

    @elpidioassinaturas
    Contributor

    Piece 2, part 1 is in #2065 (still a draft). Before it goes ready I'd like a maintainer's ok on one new dependency, since CONTRIBUTING.md asks to coordinate first:

    • schemars 1.2 (MIT; its dependencies schemars_derive, serde_derive_internals, dyn-clone, ref-cast and ref-cast-impl are MIT or Apache-2.0, all allowed by deny.toml). It generates the schema from the saved types and follows serde's own attributes, so the schema can't drift from what is saved.
    • It sits only behind an off-by-default schema feature in rustgis-geom, rustgis-data and rustgis-engine, used by one example and by cargo xtask schema. The desktop app, the browser (wasm) build, the CLI and the Python package never compile it, and THIRD-PARTY-LICENSES.txt doesn't change.

    Is that fine with you?

  12. elpidioassinaturas commented on Nov 14, 2026

    @elpidioassinaturas
    Contributor

    @giswqs @Mavrick-F, a short ping on the schemars question above, in case #2065 can make it into tonight's release: it's the only new dependency, behind an off-by-default schema feature, and the app, wasm, CLI and Python package never compile it. If you'd rather not add it, I'll hand-write the schema instead.

  13. pick2510 commented on Nov 14, 2026

    @pick2510
    ContributorAuthor

    Ok for me.

  14. Mavrick-F commented on Nov 14, 2026

    @Mavrick-F
    Member

    Fine with me, but it doesn't touch anything user facing, so don't rush it just to beat the next release.

  15. pick2510 commented on Nov 17, 2026

    @pick2510
    ContributorAuthor

    @elpidioassinaturas, a short question on piece 2: are you planning the next parts of the schema (maps and their layers, layouts and the rest that is still $defs/Open)? Piece 3 (stable IDs for layers) needs the maps and layers described, so it would help to know. If you'd rather hand it over, that's fine too, just say so here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions