Welcome! This is the community hub for the virtual-zarr organization — a place to ask questions, share ideas, and connect with other people creating and using Virtual Zarr stores.
A Virtual Zarr store lets you access existing data files (such as netCDF, HDF5, GRIB, or TIFF) through the Zarr API, without rewriting or duplicating the underlying data. Instead of copying your data into a new format, you generate lightweight references that point to the byte ranges of chunks inside the original archival files. The result is cloud-optimized, chunk-based access to data that physically stays exactly where it is.
Two projects work together to make Virtual Zarr stores practical:
-
VirtualiZarr is the tool for creating virtual Zarr stores. It lets you open archival files as virtual datasets, combine them into a single coherent datacube using familiar xarray syntax, and serialize the resulting chunk references to disk. It handles the work of pointing the Zarr data model at bytes living inside your existing files.
-
Icechunk is a cloud-native, transactional storage engine for Zarr data, and one of the formats VirtualiZarr can write its references to (the other being the Kerchunk specification). Beyond just storing references, Icechunk lets "virtual" chunks (pointing at archival files) and "native" Zarr chunks be treated interchangeably — so a store that starts out pointing entirely at archival data can be gradually extended with newly written native chunks, all with versioning and transactional guarantees.
A common workflow looks like: use VirtualiZarr to virtualize and combine your archival files, commit those references to Icechunk, and then read the data back with xarray.open_zarr as if it were a single cloud-optimized Zarr store. Questions about how these projects relate and how people use them are welcome here. A nice part about discussing on GitHub is that the conversations are persistent and findable.
This repository hosts community Discussions. It's the right place to:
- Ask questions about how to create or use Virtual Zarr stores
- Share your projects, datasets, or workflows
- Propose ideas for the virtual-zarr ecosystem
- Help others by answering questions and sharing what you've learned
Head over to the Discussions tab to start or join a conversation.
This repo is for general discussion. If you've found a bug or want to request a feature in a specific tool, please open an issue in the relevant project repository instead — for example, VirtualiZarr or Icechunk.
We're committed to providing a welcoming and inclusive environment for everyone. By participating, you agree to treat other community members with respect and kindness.