Add DASDAE storage/codec API with h5py-native compression - #817
Add DASDAE storage/codec API with h5py-native compression#817d-chambers wants to merge 1 commit into
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
📝 WalkthroughWalkthroughChangesStorage and codec flow
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderabbitai
Bot
added
IO
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
docs/tutorial/file_io.qmd (1)
59-66: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: evaluate the dict storage example.
Both the dict form and the typed form use
#| eval: false, so the render never checks them. The example patch has atimedimension with 2000 samples, so this cell can run as written. Removingeval: falsehere makes the docs fail fast if the storage contract changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/tutorial/file_io.qmd` around lines 59 - 66, Remove the #| eval: false directive from the dict-form patch.io.write example so the cell executes during documentation rendering. Keep the existing write_path, codec, compression level, and time chunk configuration unchanged.dascore/io/dasdae/utils.py (1)
191-193: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: reuse
_decode_array_valuesin_read_array_sample.
_read_array_samplerepeats the three decode branches that_decode_array_valuesnow centralizes. It can delegate and index the result, which keeps one decode contract for both readers.Proposed consolidation
def _read_array_sample(table_array, index): """Read one array sample and restore datetime-like dtypes when needed.""" - out = table_array[index] - attrs = table_array.attrs - if attrs.get("is_datetime64"): - out = np.asarray([out]).view("datetime64[ns]")[0] - if attrs.get("is_timedelta64"): - out = np.asarray([out]).view("timedelta64[ns]")[0] - if attrs.get("is_string"): - original_dtype = unbyte(attrs.get("original_string_dtype", "")) - out = convert_bytes_to_strings(np.asarray([out]), original_dtype)[0] - return out + out = np.asarray([table_array[index]]) + return _decode_array_values(out, table_array.attrs)[0]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@dascore/io/dasdae/utils.py` around lines 191 - 193, Update _read_array_sample to delegate decoding to _decode_array_values and then apply its sample/index selection to the decoded result, removing the duplicated decode branches while preserving the existing sampled-reader behavior and _read_array’s contract.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dascore/io/codec.py`:
- Around line 57-59: After the loader is executed and assigned to codec_cls on
line 58, validate that codec_cls is a subclass of BaseCodec before proceeding to
call _codec_name and register it in the registry. Skip or reject loader results
that do not meet this requirement to prevent invalid entries from being added to
the codec registry.
In `@dascore/io/dasdae/core.py`:
- Around line 88-92: Update the dims extraction in the storage validation flow
to reuse the existing normalization logic for catalog `dims` values before
calling `storage._validate_chunk_dims(all_dims)`. Parse each comma-separated
string into normalized dimension names rather than stringifying tuple-like
values, while preserving the `storage.chunks` guard and empty-scan behavior.
In `@dascore/io/dasdae/storage.py`:
- Around line 121-145: Update _resolve_chunkshape to return None whenever any
value in shape is zero, before constructing the chunk tuple. Preserve the
existing None behavior for missing chunks or mismatched dims, and leave
_dataset_options unchanged so empty arrays avoid zero-sized HDF5 chunk
dimensions.
In `@tests/test_io/test_io_core.py`:
- Around line 806-808: Update test_builtins_registered and the corresponding
assertions around the additional built-in codecs to isolate registry behavior
from installed plugins by stubbing get_entry_point_loaders() to return no
plugins. Keep the exact built-in identity assertions once plugin discovery is
disabled.
---
Nitpick comments:
In `@dascore/io/dasdae/utils.py`:
- Around line 191-193: Update _read_array_sample to delegate decoding to
_decode_array_values and then apply its sample/index selection to the decoded
result, removing the duplicated decode branches while preserving the existing
sampled-reader behavior and _read_array’s contract.
In `@docs/tutorial/file_io.qmd`:
- Around line 59-66: Remove the #| eval: false directive from the dict-form
patch.io.write example so the cell executes during documentation rendering. Keep
the existing write_path, codec, compression level, and time chunk configuration
unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1fb22d1e-4207-461e-95ca-b8fb14be4195
📒 Files selected for processing (13)
dascore/io/__init__.pydascore/io/codec.pydascore/io/core.pydascore/io/dasdae/__init__.pydascore/io/dasdae/core.pydascore/io/dasdae/storage.pydascore/io/dasdae/utils.pydascore/io/hdf5.pydocs/tutorial/file_io.qmdpyproject.tomltests/test_io/test_dasdae/test_dasdae.pytests/test_io/test_hdf5.pytests/test_io/test_io_core.py
| for loader in get_entry_point_loaders(_CODEC_ENTRY_POINT_GROUP).values(): | ||
| codec_cls = loader() | ||
| registry[_codec_name(codec_cls)] = codec_cls |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Reject loader results that are not codec classes.
Line 58 accepts any loader result. A loader that returns Gzip() or an unrelated class can produce a malformed registry or fail later outside the plugin boundary. Validate that the result is a BaseCodec subclass before calling _codec_name.
Proposed fix
for loader in get_entry_point_loaders(_CODEC_ENTRY_POINT_GROUP).values():
codec_cls = loader()
+ if not isinstance(codec_cls, type) or not issubclass(codec_cls, BaseCodec):
+ msg = "Codec entry points must return a BaseCodec subclass."
+ raise InvalidFiberIOError(msg)
registry[_codec_name(codec_cls)] = codec_cls📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for loader in get_entry_point_loaders(_CODEC_ENTRY_POINT_GROUP).values(): | |
| codec_cls = loader() | |
| registry[_codec_name(codec_cls)] = codec_cls | |
| for loader in get_entry_point_loaders(_CODEC_ENTRY_POINT_GROUP).values(): | |
| codec_cls = loader() | |
| if not isinstance(codec_cls, type) or not issubclass(codec_cls, BaseCodec): | |
| msg = "Codec entry points must return a BaseCodec subclass." | |
| raise InvalidFiberIOError(msg) | |
| registry[_codec_name(codec_cls)] = codec_cls |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@dascore/io/codec.py` around lines 57 - 59, After the loader is executed and
assigned to codec_cls on line 58, validate that codec_cls is a subclass of
BaseCodec before proceeding to call _codec_name and register it in the registry.
Skip or reject loader results that do not meet this requirement to prevent
invalid entries from being added to the codec registry.
| if self.chunks is None or not shape or len(dims) != len(shape): | ||
| return None | ||
| return tuple( | ||
| min(self.chunks.get(dim, size), size) | ||
| for dim, size in zip(dims, shape, strict=True) | ||
| ) | ||
|
|
||
| def _dataset_options(self, dims, shape) -> dict: | ||
| """ | ||
| Return h5py ``create_dataset`` options for one array. | ||
|
|
||
| Combines the codec's compression kwargs with the chunk shape resolved | ||
| for the array's dims. An empty dict means default contiguous storage. | ||
| """ | ||
| codec = self.codec | ||
| if codec is None: | ||
| out = {} | ||
| else: | ||
| # _resolve_codec guarantees only HDF5-filter codecs get this far. | ||
| assert isinstance(codec, HDF5Codec) | ||
| out = codec._dataset_kwargs() | ||
| chunkshape = self._resolve_chunkshape(dims, shape) | ||
| if chunkshape is not None: | ||
| out["chunks"] = chunkshape | ||
| return out |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
python - <<'PY'
import tempfile
import h5py
with tempfile.NamedTemporaryFile(suffix=".h5") as tmp:
with h5py.File(tmp.name, "w") as h5:
try:
h5.create_dataset("empty", shape=(0,), dtype="f8", chunks=(0,))
except Exception as exc:
print(f"Rejected zero-sized chunk dimension: {type(exc).__name__}")
else:
raise AssertionError("h5py accepted an invalid zero-sized chunk dimension")
PYRepository: DASDAE/dascore
Length of output: 269
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== candidate files =="
fd -a 'storage\.py$' . | sed 's#^\./##' | grep dascore/io/dasdae/storage.py || true
echo "== source excerpt =="
sed -n '1,220p' dascore/io/dasdae/storage.py
echo "== related references =="
rg -n "_resolve_chunkshape|_dataset_options|chunks|dset|create_dataset|HDF5Codec|storage" dascore/io/dasdae -SRepository: DASDAE/dascore
Length of output: 9532
🌐 Web query:
h5py create_dataset chunks must be positive zero values documentation
💡 Result:
In h5py, the chunks parameter for create_dataset must consist of positive integer values because it defines the shape of the data blocks stored on disk [1][2][3]. HDF5 requires that chunk dimensions be strictly greater than zero; a value of zero in any dimension of the chunk shape will result in a ValueError with the message, "All chunk dimensions must be positive" [4][5][6]. This requirement is enforced by the underlying HDF5 library's property list settings [4][7][5]. When specifying chunks manually, you must provide a tuple of positive integers that matches the rank (number of dimensions) of the dataset [1][2][7]. Key points regarding this requirement: - Chunked storage relies on dividing a dataset into regular, fixed-size pieces [2][8]. A dimension of zero is logically incompatible with the definition of a fixed-size chunk [4][6]. - If you encounter this error, ensure that none of the values in the tuple passed to the chunks argument are 0 or negative [4][5]. - If you are working with datasets that involve dynamic or unlimited dimensions, you should use None for the unlimited axes in the shape or maxshape parameters, but the chunk size for any active dimension must still be a positive integer [9][4]. - If you are unsure what chunk shape to use, you can pass chunks=True to allow h5py to automatically determine an appropriate chunk shape for you [1][8].
Citations:
- 1: https://docs.h5py.org/en/stable/high/dataset.html?highlight=chunking
- 2: https://docs.h5py.org/en/3.15.1/high/dataset.html
- 3: https://docs.h5py.org/en/latest/high/dataset.html
- 4: Cannot read zero size datasets to NumPy array h5py/h5py#281
- 5: All chunk dimensions must be positive velocyto-team/velocyto.py#92
- 6: Cannot set a dataset to a scalar PDLPorters/pdl-io-hdf5#9
- 7: https://github.com/h5py/h5py/blob/c2ad0b91f074b5b62d5c10b3970d39ae55b8ec1f/h5py/h5p.pyx
- 8: https://docs.h5py.org/en/3.11.0/high/dataset.html
- 9: https://docs.h5py.org/en/stable/high/group.html
Do not emit zero-sized chunk dimensions.
_resolve_chunkshape(dims, shape=(..., 0, ...)) currently returns a chunk tuple such as (..., 0, ...), while HDF5 dataset chunks must be strictly positive integers. Return None when any array dimension is empty so the existing empty-array write path can use contiguous storage or automatic chunking.
Proposed fix
- if self.chunks is None or not shape or len(dims) != len(shape):
+ if (
+ self.chunks is None
+ or not shape
+ or any(size == 0 for size in shape)
+ or len(dims) != len(shape)
+ ):
return None📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if self.chunks is None or not shape or len(dims) != len(shape): | |
| return None | |
| return tuple( | |
| min(self.chunks.get(dim, size), size) | |
| for dim, size in zip(dims, shape, strict=True) | |
| ) | |
| def _dataset_options(self, dims, shape) -> dict: | |
| """ | |
| Return h5py ``create_dataset`` options for one array. | |
| Combines the codec's compression kwargs with the chunk shape resolved | |
| for the array's dims. An empty dict means default contiguous storage. | |
| """ | |
| codec = self.codec | |
| if codec is None: | |
| out = {} | |
| else: | |
| # _resolve_codec guarantees only HDF5-filter codecs get this far. | |
| assert isinstance(codec, HDF5Codec) | |
| out = codec._dataset_kwargs() | |
| chunkshape = self._resolve_chunkshape(dims, shape) | |
| if chunkshape is not None: | |
| out["chunks"] = chunkshape | |
| return out | |
| if ( | |
| self.chunks is None | |
| or not shape | |
| or any(size == 0 for size in shape) | |
| or len(dims) != len(shape) | |
| ): | |
| return None | |
| return tuple( | |
| min(self.chunks.get(dim, size), size) | |
| for dim, size in zip(dims, shape, strict=True) | |
| ) | |
| def _dataset_options(self, dims, shape) -> dict: | |
| """ | |
| Return h5py ``create_dataset`` options for one array. | |
| Combines the codec's compression kwargs with the chunk shape resolved | |
| for the array's dims. An empty dict means default contiguous storage. | |
| """ | |
| codec = self.codec | |
| if codec is None: | |
| out = {} | |
| else: | |
| # _resolve_codec guarantees only HDF5-filter codecs get this far. | |
| assert isinstance(codec, HDF5Codec) | |
| out = codec._dataset_kwargs() | |
| chunkshape = self._resolve_chunkshape(dims, shape) | |
| if chunkshape is not None: | |
| out["chunks"] = chunkshape | |
| return out |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@dascore/io/dasdae/storage.py` around lines 121 - 145, Update
_resolve_chunkshape to return None whenever any value in shape is zero, before
constructing the chunk tuple. Preserve the existing None behavior for missing
chunks or mismatched dims, and leave _dataset_options unchanged so empty arrays
avoid zero-sized HDF5 chunk dimensions.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #817 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 176 179 +3
Lines 19372 19589 +217
==========================================
+ Hits 19372 19589 +217
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
7922eb5 to
c747d02
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dascore/io/netcdf/core.py`:
- Around line 86-98: The _get_write_encoding method must reject compression_opts
when compression is not "gzip", and validate gzip compression_opts as an integer
level from 1 through 9 before constructing encoding. Preserve the default level
of 4 when compression_opts is None, and add boundary tests covering None, 0, 10,
and options supplied without gzip.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 9a95a69e-7fdc-4f19-8b4e-bd6b4d964bf7
📒 Files selected for processing (9)
dascore/io/codec.pydascore/io/core.pydascore/io/dasdae/core.pydascore/io/netcdf/core.pydocs/tutorial/file_io.qmdpyproject.tomltests/test_io/test_dasdae/test_dasdae.pytests/test_io/test_io_core.pytests/test_io/test_prodml/test_prodml_write.py
🚧 Files skipped from review as they are similar to previous changes (6)
- pyproject.toml
- docs/tutorial/file_io.qmd
- dascore/io/dasdae/core.py
- tests/test_io/test_io_core.py
- tests/test_io/test_dasdae/test_dasdae.py
- dascore/io/core.py
| def _get_write_encoding(self, compression=None, compression_opts=None, chunks=None): | ||
| """Translate explicit write options into xarray encoding hints.""" | ||
| compression = kwargs.get("compression") | ||
| if compression not in ("gzip", None, False): | ||
| msg = "xarray netcdf4 writing currently supports only gzip compression." | ||
| raise ValueError(msg) | ||
| chunks = kwargs.get("chunks") | ||
| encoding: dict[str, object] = {} | ||
| if chunks not in (None, False, True): | ||
| encoding["chunksizes"] = tuple(chunks) | ||
| if compression == "gzip": | ||
| encoding["zlib"] = True | ||
| encoding["complevel"] = kwargs.get("compression_opts", 4) | ||
| encoding["complevel"] = 4 if compression_opts is None else compression_opts | ||
| encoding["shuffle"] = True | ||
| return encoding |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- source outline ---'
ast-grep outline dascore/io/netcdf/core.py || true
echo '--- source lines ---'
cat -n dascore/io/netcdf/core.py | sed -n '1,140p'
cat -n dascore/io/netcdf/core.py | sed -n '200,240p'
echo '--- relevant references ---'
rg -n --glob '!*.lock' --glob '!uv.lock' \
'compression_opts|_get_write_encoding|to_netcdf|h5netcdf|netcdf4|complevel|chunksizes' .
echo '--- dependency/config files ---'
git ls-files | rg '(^|/)(pyproject\.toml|setup\.cfg|setup\.py|requirements[^/]*|.*lock|tox\.ini|environment[^/]*)$' | sortRepository: DASDAE/dascore
Length of output: 13163
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- focused NetCDF tests ---'
cat -n tests/test_io/test_netcdf/test_netcdf.py | sed -n '270,325p'
cat -n tests/test_io/test_netcdf/test_netcdf.py | sed -n '740,785p'
cat -n tests/test_io/test_netcdf/test_netcdf.py | sed -n '390,475p'
echo '--- project dependency constraints ---'
cat -n pyproject.toml | sed -n '1,110p'
cat -n environment.yml | sed -n '1,140p'
echo '--- IO option definitions and docs ---'
rg -n -C 4 \
'compression_opts|compression=|chunks=|write\(' \
dascore tests docs README.md 2>/dev/null | sed -n '1,260p'Repository: DASDAE/dascore
Length of output: 36555
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- IO API definitions ---'
ast-grep outline dascore/io/core.py || true
rg -n -C 8 'def write|compression_opts|compression|chunks' dascore/io dascore/*.py | sed -n '1,260p'
echo '--- lock/version files ---'
git ls-files | rg '(^|/)(uv\.lock|poetry\.lock|Pipfile\.lock|environment.*\.ya?ml|pyproject\.toml)$' | sort
if git ls-files --error-unmatch uv.lock >/dev/null 2>&1; then
rg -n '(^name = "(xarray|h5netcdf|netCDF4)"$|^version = )' uv.lock | sed -n '1,120p'
fiRepository: DASDAE/dascore
Length of output: 20069
🌐 Web query:
xarray Dataset.to_netcdf encoding zlib complevel h5netcdf compression_opts valid range gzip level documentation
💡 Result:
When writing a Dataset to a NetCDF file using the xarray to_netcdf method, you can specify compression settings via the encoding parameter [1][2]. The encoding argument is a nested dictionary where keys are variable names and values are dictionaries of specific encoding options for that variable [1][2]. Providing this dictionary causes xarray to ignore the original encoding of the variables [1][2]. Engine Support and Compression Parameters The h5netcdf engine, which is commonly used with xarray, supports two primary styles of compression encoding [1][2]: 1. NetCDF4-style (compatible with the netcdf4 library): Use zlib and complevel. Example: {"zlib": True, "complevel": 9} [1][2]. complevel is typically an integer from 0 to 9, where higher numbers generally indicate greater compression [3]. 2. h5py-style (h5py native): Use compression and compression_opts. Example: {"compression": "gzip", "compression_opts": 9} [1][2]. The compression parameter is set to "gzip" (or other available HDF5 compression filters), and compression_opts controls the filter parameters, such as the GZIP compression level (0-9) [3]. Important Considerations - Overriding Encoding: Using the encoding parameter completely replaces any existing encoding for the specified variables [1][2]. If you intend to retain existing settings like _FillValue, you must explicitly include them in your encoding dictionary [4]. - Compatibility: The h5netcdf engine is designed to handle both styles, providing flexibility when working with different HDF5/NetCDF workflows [1][2]. - Engine Defaults: Depending on your environment, the default engine may vary (e.g., netcdf4, h5netcdf, or scipy) [1][2]. You can verify or set the engine explicitly using the engine parameter in to_netcdf or manage the default order via xarray.set_options(netcdf_engine_order=...) [1][2]. For further details on specific compression levels or custom filters, refer to the underlying h5py documentation regarding HDF5 datasets [3].
Citations:
- 1: https://docs.xarray.dev/en/stable/generated/xarray.Dataset.to_netcdf.html
- 2: https://docs.xarray.dev/en/v2026.04.0/generated/xarray.Dataset.to_netcdf.html
- 3: https://docs.h5py.org/en/3.15.0/high/dataset.html
- 4: Compression & to_netcdf pydata/xarray#5709
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import ast
from pathlib import Path
path = Path("dascore/io/netcdf/core.py")
tree = ast.parse(path.read_text())
method = next(
node for node in ast.walk(tree)
if isinstance(node, ast.FunctionDef) and node.name == "_get_write_encoding"
)
# Inspect the method as data, then model only its literal branch assignments.
source = ast.get_source_segment(path.read_text(), method)
has_range_guard = any(
isinstance(node, ast.Compare)
and any(isinstance(op, (ast.Lt, ast.Gt, ast.LtE, ast.GtE)) for op in node.ops)
for node in ast.walk(method)
)
has_no_gzip_guard = any(
isinstance(node, ast.If)
and isinstance(node.test, ast.Compare)
and any(
isinstance(comparator, ast.Constant)
and comparator.value == "gzip"
for comparator in node.test.comparators
)
for node in ast.walk(method)
)
def current_encoding(compression, compression_opts):
encoding = {}
if compression == "gzip":
encoding.update(
zlib=True,
complevel=4 if compression_opts is None else compression_opts,
shuffle=True,
)
return encoding
print(f"range_guard_present={has_range_guard}")
print(f"gzip_branch_present={has_no_gzip_guard}")
for compression, opts in [
("gzip", None),
("gzip", 0),
("gzip", 10),
(None, 5),
(False, 5),
]:
print((compression, opts), current_encoding(compression, opts))
PYRepository: DASDAE/dascore
Length of output: 415
Reject incompatible and out-of-range compression options.
When compression_opts is set without compression="gzip", raise ValueError instead of silently ignoring it. When gzip is selected, enforce the documented 1-9 range before creating encoding. Add boundary tests for None, 0, 10, and compression_opts without gzip.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@dascore/io/netcdf/core.py` around lines 86 - 98, The _get_write_encoding
method must reject compression_opts when compression is not "gzip", and validate
gzip compression_opts as an integer level from 1 through 9 before constructing
encoding. Preserve the default level of 4 when compression_opts is None, and add
boundary tests covering None, 0, 10, and options supplied without gzip.
Source: MCP tools
1 participant
Description
Adds a storage/codec API for DASDAE writes, ported to dev's h5py-based IO stack. This supersedes #734, which was written against master's PyTables implementation.
Codecs describe how array payloads are compressed; storage objects describe how a writer applies those codecs plus layout options such as per-dimension chunking. Keeping the two separate means chunk layout stays a storage concern and codecs remain reusable by other HDF5-backed formats.
User API
Simple preset compression:
Dict form with a codec and chunk layout (no imports needed):
Typed form:
Capability discovery:
Implementation
BaseCodecandBaseStoragepydantic models indascore.io.core, plusget_storage()/get_codecs()discovery.FiberIO.storage_clsis derived from thestorageannotation onwrite()so the storage type has a single source of truth.dascore.codecentry-point group) indascore.io.codec; onlyget_codecsis exported on thedascore.ionamespace to avoid aget_codec/get_codecsnaming trap.Gzipcodec indascore.io.hdf5. Blosc/zstd is not included for now: h5py has no built-in blosc filter, so thecompressedpreset uses gzip level 5. A futurehdf5plugin-backed codec can restore it through the registry without API changes.DASDAEStoragewithcodec/chunksoptions and fail-fast validation: unknown codec names, codec instances without a registered discriminator, non-positive chunk sizes, and typoed chunk dimension names all raise before any data is written. Chunk-dim validation uses scan metadata so lazy spools are not materialized twice.Compatibility
Files written without a storage argument are byte-for-byte equivalent to before. Compressed files use native HDF5 filters, so any HDF5 reader (h5py, PyTables, external tools) can read them without DASCore.
Checklist
I have (if applicable):
Summary by CodeRabbit
New Features
Bug Fixes
Documentation