Transition to new dataspec

[tristan-deep]

Initial rewrite of dataspec is introduced in #307. Development of this issue is ongoing and will be merged to the branch openh-rf

High priority

• Check validation workflow on https://github.com/open-h/openh-rf-dev
• Add legacy file support and add test for this. @swpenninga Add legacy data format support for v0.1.0 #368
  • After all conversion scripts have been updated, also upload legacy dataset to huggingface branch v0.0.13 and new to v0.1.0. See below for assignments.
  • After merge and release, remove all revision=v0.1.0 tags again from zea.File in snippets / examples.
• Check the conversion examples (PICMUS etc.) and make sure we point to them in the docs / OpenH-RF page.
  • PICMUS / CETUS / CAMUS @tristan-deep Extended ProbeSpec #379 Updated conversion scripts (PICMUS, CAMUS, CETUS) #385
  • DehazingEcho2025 @tristan-deep
  • EchoNetLVH @wesselvannierop
  • EchoNetDynamic @swpenninga
  • Carotid @wesselvannierop
  • Verasonics @wesselvannierop
  • Phantoms @OisinNolan

Medium priority

• Improve and clean up docs regarding data handling (zea.File, zea.Dataset, zea.Dataloader) and dataformat itself.
  • General improvements
  • Add doctest to getting-started.rst (see New zea dataspec #307 (comment))
• Remove outdated data types figure / diagram and documentation in docs.
• Remove image_sc where possible (deprecated data type).
• Support for some custom parameters, for research purposes (#New zea dataspec #307 (comment))
• Consider using lzf instead of gzip for more speed during loading/saving at the cost of larger filesize Consider lzf compression for zea.File instead of gzip #396 @tristan-deep Changed compression default to lzf #399
• Remove old Config schema (validation.py) and convert to a light ConfigSpec to be more consistent with ProbeSpec and ScanSpec (mainly inherits from these as well for config.parameters)
• Rename config.rst to parameters.rst and create a docs page centered around the whole concept of parameters not just config (see here). Can remove the config parameter table there as well as this defined mostly by the probe and scan spec which are on the data-acquisition.rst already (refer to those).
• Clean up warnings on saving, we don't want to flood the user with every possible key in the data format: Clean up data loading and saving warnings #328

Low priority

• Making zea.data.generate_zea_dataset more robust #281
• Check all generate_zea_dataset and remove where possible (use real legacy files for legacy compatibility instead, see convo).
• Update zea.File to support all features in file_operations.py module.
• Remove Interface class and update with CLI (move to zea.main?) Introduce a beamforming CLI #372
• Should we open the file at the end of File.create? Do not open file at the end of File.create #410
• Rename scan.py -> parameters.py
• Clean up tests, group in folders etc...

Completed

• Make properties of file.scan, file.probe and file.metadata (like file.data) Extended ProbeSpec #379
• Maps like Segmentations should be allowed to be 2d, e.g. (frames, n_z, n_x). Currently this fails validation, even though the docstring says it's supported. See gist @swpenninga
• Improve repr methods on all classes related to zea.File (e.g. DataSpec, Scan, Probe, Tracks, etc.). This will make "browsing" through a file in debug mode much more friendly. @tristan-deep Improve repr / str across zea objects #386
• Checkout if we should store data as complex instead of with two channels (advantages/disadvantages) Complex to channels and back now in Keras #291
• Remove load_file from all the example scripts (replace with zea.File context manager) Updated conversion scripts (PICMUS, CAMUS, CETUS) #385
• zea.data.convert.matlab.read_raw_data() requires fix for nr frames is one #112 @wesselvannierop
• Improve docs regarding pipeline creation. This will be part of the submission process and important for validating the submitted files. @tristan-deep Improved pipeline docs #384
• Include aligned_data as spatial maps @swpenninga Make aligned data spatial map & Allow 2D segmentations #383
• Adding probespec Add ProbeSpec #363 @tristan-deep Extended ProbeSpec #379
• Map coordinates should support broadcasting, i.e. allow coordinates to have 1 dimension less at the start, but otherwise match. @OisinNolan Allow broadcasting of Map.coordinates #380
• Fix rabbit / review comments
• Clean up data loading and saving warnings #328 @swpenninga Add warnings for optional scan fields, fix nested warnings #361
• Consider difference image and image_sc, and their extent (polar vs cartesian) @OisinNolan Move Maps from extent -> coordinates #362
• Related to above. Look into extents. xlims, ylims, zlims, extent, polar_lims. Do we save extent type with the data? Improve docs on this as well. @OisinNolan Move Maps from extent -> coordinates #362
• Ensure Map validation warning messages point to specific map type @OisinNolan
• Implement Track abstraction to support multiple acquisition schemes in one file @OisinNolan Acquisition tracks in DataSpec #359

[mxncr]

Hi, thanks for your work on standardizing ultrasound dataset formats. We (Resolve Stroke) will probably contribute to the OpenH-RF dataset.

On data storage, have you considered using Zarr instead of, or in addition to, HDF5? It is a more modern format for n-dimensional array storage with significant advantages over HDF5, notably support for concurrent read/write and cloud streaming. The hierarchical data organization is mostly the same.

Happy to discuss further if you are interested.

[tristan-deep]

Hi @mxncr thanks for showing interest! And great to hear Resolve Stroke is also interested in contributing.

We did actually consider Zarr (and some other data formats like Arrow) and personally I do see the benefits. In the end a practical choice was made to only change extend / update the dataspec (for OpenH-RF) and not the dataformat itself (so keep HDF5). Also given the work it would take to migrate and the approaching OpenH-RF deadline. We did some experimental testing with Zarr and saw only limited advantages with respect to HDF5 given our requirements. Also, the fact that HDF5 is quite established in the community (wide support) and is only a single binary file appealed quite a bit to us.

That said, it does look promising, and we might add some support in the future (post OpenH-RF) in case it really shows some benefits. I can imaging that in some ML workflows there would be a good advantage. What are your experiences?

[mxncr]

Yes, totally understandable, file format migration is a significant endeavour.
In our case we did migrate from HDF5 to Zarr v3 last year and are very happy with it. Our context is a bit particular though: we work with large acquisitions (200–300 GB each) and were hitting some painful HDF5 limitations, mainly the inability to do concurrent reads/writes. The ability to lazily access remote datasets and retrieve only the necessary chunks (e.g. via Dask) has also been very convenient.