Add support for zarr

[robinskil]

This pull request introduces a new crate, beacon-arrow-zarr, to the workspace, providing foundational support for reading and streaming Zarr array data with Arrow interoperability. The update includes new Rust modules for handling Zarr data types, array slicing, attributes, and streaming, as well as the addition of test fixtures for Zarr arrays. There are also minor dependency updates and workspace configuration changes.

New crate and Zarr/Arrow interoperability:

• Added beacon-arrow-zarr crate to the workspace, with its own Cargo.toml and dependencies for Zarr, Arrow, and related libraries. [1] [2]
• Implemented core modules in beacon-arrow-zarr:
  • array_slice_pushdown.rs: Defines ArraySlicePushDown for efficient dimension-based array slicing.
  • attributes.rs: Handles conversion between JSON attributes and Arrow data types.
  • data_types.rs: Maps Zarr data types to Arrow data types.
  • stream.rs: Provides an async stream interface (ArrowZarrStreamComposer, ArrowZarrStream) for reading Zarr data as Arrow batches, including support for chunking and array slicing.
  • lib.rs: Exposes the new modules.

Test data and fixtures:

• Added several Zarr metadata JSON files under beacon-arrow-zarr/test_files/gridded-example.zarr/ for arrays like analysed_sst, analysis_error, lat, and lon. These serve as test fixtures for the new Zarr reading functionality. [1] [2] [3] [4]

Dependency and configuration updates:

• Updated the workspace and main Cargo.toml:
  • Added beacon-arrow-zarr to the workspace members.
  • Loosened the serde dependency version constraint and added crossbeam as a workspace dependency.

These changes lay the groundwork for robust Zarr array support in the project, enabling efficient, Arrow-based access to multi-dimensional scientific data.

[Copilot]

Pull Request Overview

This pull request introduces comprehensive Zarr array support to the project by adding a new beacon-arrow-zarr crate and integrating it with the existing DataFusion-based file format system. The implementation provides efficient streaming access to Zarr data with Arrow interoperability, predicate pushdown capabilities, and support for CF conventions.

• Adds new beacon-arrow-zarr crate with async streaming, attribute handling, and data type conversion
• Integrates Zarr format into the existing file format framework with DataFusion support
• Implements array slicing and predicate pushdown for efficient query processing

Reviewed Changes

Copilot reviewed 27 out of 65 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
beacon-arrow-zarr/Cargo.toml	New crate configuration with Zarr, Arrow, and async dependencies
beacon-arrow-zarr/src/*.rs	Core Zarr-to-Arrow conversion modules including streaming, attributes, and data types
beacon-arrow-zarr/test_files/	Test fixtures with Zarr metadata JSON files for testing
beacon-formats/src/zarr/	DataFusion integration modules for Zarr file format support
beacon-formats/src/lib.rs	Registration of new Zarr format factory
beacon-data-lake/src/table/	Table format extensions to support Zarr file format
Cargo.toml	Workspace updates to include new crate and dependency changes

Comments suppressed due to low confidence (1)

beacon-formats/src/zarr/mod.rs:1

• Typo in comment: 'reprensents' should be 'represents'.

use std::{

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}