Unable to run `igneous mesh forge` on OME-Zarr (v0.4) labels

[LucaMarconato]

TLDR; (very quick to reproduce)

1. Download 20200812-CardiomyocyteDifferentiation 14-Cycle1.zarr.zip from https://zenodo.org/records/10257532 (700 MB)
2. Unzip
3. Run

cd 20200812-CardiomyocyteDifferentiation14-Cycle1.zarr/B/03/0/labels/nuclei
igneous mesh forge . --queue my-queue && igneous execute -x -n 8 my-queue

This will fail with

  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/datasource/zarr/image.py", line 55, in decode_chunk
    raise exceptions.EmptyVolumeException(f"{filename} is missing.")
cloudvolume.exceptions.EmptyVolumeException: 0/0/3/0 is missing.

In the next I add extra details that may help in attacking the problem.

The problem

I'm trying to use igneous to generate meshes in the Neuroglancer precomputed format from OME-Zarr (v0.4) data but I am getting some errors. I'm not sure if I found some bugs or if I'm doing something wrong.

I'll explain how to reproduce but I will also describe a bit the dataset I'm using since it could make it easier to understand what's the problem here.

The dataset

From the cloudvolume code for (OME-)Zarr metadata I see that cloudvolume expects a OME-Zarr file that follows the OME-NGFF 0.4 specification. So I looked for a small OME-Zarr 3D datasets containing labels: https://zenodo.org/records/10257532 and in the 0.4 format.

I verified that the data is a valid OME-Zarr dataset by downloading it, running http-server on it and by using the OME-NGFF validator. Here I show two screenshots, one for the image part and one for the label part (nuclei):

Here below is a visualization using neuroglancer of the labels part of the data (you can obtain it by pressing the neuroglancer icon from the NGFF validator webpage).

Here instead is a GIF of exploring the data using napari-ome-zarr. As you can see the label indexing is consistent across the Z dimension, so I would expect a meshing algorithm to be able to reconstruct meshes.

A problem with the data shape?

Using this code

##
# load the data with cloudvolume
from cloudvolume.datasource.zarr import create_zarr
import numpy as np

# this is the data downloaded before
PATH = '20200812-CardiomyocyteDifferentiation14-Cycle1.zarr/B/03/0'

image = create_zarr(cloudpath=PATH)
print(image)

labels = create_zarr(cloudpath=PATH + '/labels/nuclei')
print(labels)

##
# Check the shapes of the image and labels object
print(image.volume_size)
print(labels.volume_size)

print(image.shape)
print(labels.shape)

I get

[5120 4320   10]
[1280 1080   10]
(5120, 4320, 10, 1)  # <------ why "1" and not "3"?
(1280, 1080, 10, 1)  # <------ is the "1" wanted here?

Please notice how the OME-NGFF validator tells that the shape of the labels (zyx) is [10, 1080, 1280] and that the shape of the image (czyx) is [3,10,4320,5120].

Here I see that the shape of the labels from clouddata has an extra 1. Maybe this is the cause of the problem that I am experiencing later. Also please see that the image has a 1 instead of a 3. Anyway, I'm not using the image for computing the meshes, so this is not related to this issue.

Running igneous mesh

When I use

cd 20200812-CardiomyocyteDifferentiation14-Cycle1.zarr/B/03/0/labels/nuclei
igneous mesh forge . --queue my-queue && igneous execute -x -n 8 my-queue

I get the error

  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/datasource/zarr/image.py", line 55, in decode_chunk
    raise exceptions.EmptyVolumeException(f"{filename} is missing.")
cloudvolume.exceptions.EmptyVolumeException: 0/0/3/0 is missing.

Full output and traceback

Inserting: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 70.90it/s]
INFO Running MeshTask(shape=[448, 448, 448],offset=[0, 896, 0],layer_path='file:///Users/macbook/ssd/biodata/ome-zarr/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr/B/03/0/labels/nuclei',mip=0,simplification_factor=100,max_simplification_error=40,mesh_dir='mesh_mip_0_err_40',cache_control='no-cache',dust_threshold=None,dust_global=False,progress=False,object_ids=None,fill_missing=False,encoding='precomputed',spatial_index=True,frag_path=None,sharded=False,compress='gzip',closed_dataset_edges=True,fill_holes=0,dry_run=False)  (id: f2da9b69-1269-4bd4-9153-d837d7b56a6e)
ERROR MeshTask(shape=[448, 448, 448],offset=[0, 896, 0],layer_path='file:///Users/macbook/ssd/biodata/ome-zarr/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr/B/03/0/labels/nuclei',mip=0,simplification_factor=100,max_simplification_error=40,mesh_dir='mesh_mip_0_err_40',cache_control='no-cache',dust_threshold=None,dust_global=False,progress=False,object_ids=None,fill_missing=False,encoding='precomputed',spatial_index=True,frag_path=None,sharded=False,compress='gzip',closed_dataset_edges=True,fill_holes=0,dry_run=False) raised 0/0/3/0 is missing.
 Traceback (most recent call last):
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/taskqueue/taskqueue.py", line 378, in poll
    task.execute(*execute_args, **execute_kwargs)
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/igneous/tasks/mesh/mesh.py", line 177, in execute
    data = self._volume.download(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/frontends/precomputed.py", line 818, in download
    tup = self.image.download(
          ^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/datasource/zarr/image.py", line 149, in download
    chunk = self.decode_chunk(binary, mip, fname, self.meta.zarr_chunk_size(mip))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/datasource/zarr/image.py", line 55, in decode_chunk
    raise exceptions.EmptyVolumeException(f"{filename} is missing.")
cloudvolume.exceptions.EmptyVolumeException: 0/0/3/0 is missing.

Traceback (most recent call last):
  File "/opt/miniconda3/envs/cv/bin/igneous", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/click/decorators.py", line 34, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/igneous_cli/cli.py", line 880, in execute
    parallel_execute_helper(parallel, args)
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/igneous_cli/cli.py", line 884, in parallel_execute_helper
    execute_helper(*args)
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/igneous_cli/cli.py", line 919, in execute_helper
    tq.poll(
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/taskqueue/taskqueue.py", line 378, in poll
    task.execute(*execute_args, **execute_kwargs)
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/igneous/tasks/mesh/mesh.py", line 177, in execute
    data = self._volume.download(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/frontends/precomputed.py", line 818, in download
    tup = self.image.download(
          ^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/datasource/zarr/image.py", line 149, in download
    chunk = self.decode_chunk(binary, mip, fname, self.meta.zarr_chunk_size(mip))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/cv/lib/python3.12/site-packages/cloudvolume/datasource/zarr/image.py", line 55, in decode_chunk
    raise exceptions.EmptyVolumeException(f"{filename} is missing.")
cloudvolume.exceptions.EmptyVolumeException: 0/0/3/0 is missing.

Which may suggest that the extra dimension "1" in the shape of the labels is indeed the problem, since it seems that igneous is trying to access a chunk that doesn't exist?

Any help is appreciated!

CC @EricMoerthVis

[william-silversmith]

Hi! My implementation of zarr is homegrown and limited.... I have been expanding its capabilities as people request them. I'm currently working on adding support for unsharded zarr3. I can take a look into this next week though.

However, to my (perhaps outdated?) knowledge, Neuroglancer only supports meshes in the Precomputed format. If that's not true, then I'll have to add support for zarr specifically.

I would recommend converting this data to precomputed. Since this dataset is only 1.3GB, it should be possible to use zarr to load it as a numpy array in memory and then convert it to Precomputed using CloudVolume.from_numpy. Then you can apply igneous meshing at will to the dataset on disk.

[LucaMarconato]

Thank you for your answer. Happy to share some updates regarding mesh support in NGFF/OME-Zarr (which gives a broader context to what is behind this tissue). In this discussion it is being proposed to adopt the neuroglancer format for meshes within NGFF. A draft of the specs is available here.

I am currently exploring libraries/code snippets to read/write mesh data in the new multilod precomputed mesh format. I checked this code and this other one (both they do not support sharding) and now In am experimenting with igneous and cloudvolume.

My objective is to draft a workflow in which one can create meshes from OME-Zarr data without converting the images to the precomputed format and see if and where the friction points are. If you could have a look at this issue it would be great thank you, I will also see if I can find the cause of the error 😊

[kephale]

@william-silversmith if you have opinions on how we implement multiscale meshes in NGFF, we'd love to have you chime in on the discussion as well 🙏

[william-silversmith]

Thanks! I think I'll have more of an opinion once I get to the point of (a) implementing sharded images in zarr3 and (b) thinking about the meshing components more carefully. I'm currently working on zarr3 unsharded images right now. I need to have a firmer grasp of the differences in sharding implementations for one.