Proseg generates a "broken" file for points?

[sshen82]

Hi,
Thank you for this great tool! I am trying to use Proseg on my Xenium data, but in the "points" folder, it gives a part.0.parquet file. Because of this file, SpatialData cannot load it. I also can't convert this whole thing into the format that Xenium Explorer can use. What should I check?

(Proseg) sshen2@maestro:/fh/working/sun_w/TRPCD/RP1_ANT-12__20250806__201253$ proseg --xenium transcripts.parquet -t 10
Using 10 threads
Finished reading input
Read dataset:
239331881 transcripts
456085 cells
5050 genes
478 fovs

00:00:00 ############################################################ | log-likelihood: -1146360600 | assigned: 238811296 / 239331881 (99.78%) | non-background: (99.72%)

[dcjones]

Can you show me how you are trying to read the output and what kind of error message you get?

[sshen82]

Sure, I try to do adata_och35_proseg = spatialdata.read_zarr("/fh/working/sun_w/TRPCD/RP10_OCH-35__20260120__213109/proseg-output.zarr"), but below is the error.

/home/sshen2/.conda/envs/GPU-SP/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
/tmp/ipykernel_1148606/1120284084.py:35: UserWarning: SpatialData is not stored in the most current format. If you want to use Zarr v3, please write the store to a new location using sdata.write().
adata_och35_proseg = spatialdata.read_zarr("/fh/working/sun_w/TRPCD/RP10_OCH-35__20260120__213109/proseg-output.zarr")
/home/sshen2/.conda/envs/GPU-SP/lib/python3.12/site-packages/zarr/core/group.py:3535: ZarrUserWarning: Object at .DS_Store is not recognized as a component of a Zarr hierarchy.
warnings.warn(

ArrowInvalid Traceback (most recent call last)
File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/backends.py:140, in CreationDispatch.register_inplace..decorator..wrapper(*args, **kwargs)
139 try:
--> 140 return func(*args, **kwargs)
141 except Exception as e:

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/dataframe/dask_expr/_collection.py:5384, in read_parquet(path, columns, filters, categories, index, storage_options, dtype_backend, calculate_divisions, ignore_metadata_file, metadata_task_size, split_row_groups, blocksize, aggregate_files, parquet_file_extension, filesystem, engine, arrow_to_pandas, **kwargs)
5365 return new_collection(
5366 ReadParquetPyarrowFS(
5367 path,
(...) 5380 )
5381 )
5383 return new_collection(
-> 5384 ReadParquetFSSpec(
5385 path,
5386 columns=_convert_to_list(columns),
5387 filters=filters,
5388 categories=categories,
5389 index=index,
5390 storage_options=storage_options,
5391 calculate_divisions=calculate_divisions,
5392 ignore_metadata_file=ignore_metadata_file,
5393 metadata_task_size=metadata_task_size,
5394 split_row_groups=split_row_groups,
5395 blocksize=blocksize,
5396 aggregate_files=aggregate_files,
5397 parquet_file_extension=parquet_file_extension,
5398 filesystem=filesystem,
5399 engine=_set_parquet_engine(engine),
5400 kwargs=kwargs,
5401 _series=isinstance(columns, str),
5402 )
5403 )

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/_expr.py:873, in SingletonExpr.new(cls, _determ_token, *args, **kwargs)
872 cls._instances = weakref.WeakValueDictionary()
--> 873 inst = super().new(cls, *args, _determ_token=_determ_token, **kwargs)
874 _name = inst._name

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/_expr.py:69, in Expr.new(cls, _determ_token, *args, **kwargs)
67 # This is typically cached. Make sure the cache is populated by calling
68 # it once
---> 69 inst._name
70 return inst

File ~/.conda/envs/GPU-SP/lib/python3.12/functools.py:995, in cached_property.get(self, instance, owner)
994 if val is _NOT_FOUND:
--> 995 val = self.func(instance)
996 try:

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/dataframe/dask_expr/io/parquet.py:790, in ReadParquet._name(self)
788 @cached_property
789 def _name(self):
--> 790 return f"{self._funcname}-{self.deterministic_token}"

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/_expr.py:536, in Expr.deterministic_token(self)
533 if not self._determ_token:
534 # Just tokenize self to fall back on dask_tokenize
535 # Note how this differs to the implementation of dask_tokenize
--> 536 self._determ_token = self.dask_tokenize()
537 return self._determ_token

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/dataframe/dask_expr/io/parquet.py:784, in ReadParquet.dask_tokenize(self)
781 if not self._determ_token:
782 # TODO: Is there an actual need to overwrite this?
783 self._determ_token = _tokenize_deterministic(
--> 784 funcname(type(self)), self.checksum, *self.operands[:-1]
785 )
786 return self._determ_token

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/dataframe/dask_expr/io/parquet.py:794, in ReadParquet.checksum(self)
792 @Property
793 def checksum(self):
--> 794 return self._dataset_info["checksum"]

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/dataframe/dask_expr/io/parquet.py:1377, in ReadParquetFSSpec._dataset_info(self)
1358 args = (
1359 paths,
1360 fs,
(...) 1375 },
1376 )
-> 1377 dataset_info = self.engine._collect_dataset_info(*args)
1378 checksum = []

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py:991, in ArrowDatasetEngine._collect_dataset_info(cls, paths, fs, categories, index, gather_statistics, filters, split_row_groups, blocksize, aggregate_files, ignore_metadata_file, metadata_task_size, parquet_file_extension, kwargs)
990 if ds is None:
--> 991 ds = pa_ds.dataset(
992 paths,
993 filesystem=_wrapped_fs(fs),
994 **_processed_dataset_kwargs,
995 )
997 # Get file_frag sample and extract physical_schema

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/pyarrow/dataset.py:793, in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
792 if all(_is_path_like(elem) or isinstance(elem, FileInfo) for elem in source):
--> 793 return _filesystem_dataset(source, **kwargs)
794 elif all(isinstance(elem, Dataset) for elem in source):

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/pyarrow/dataset.py:482, in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
480 factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
--> 482 return factory.finish(schema)

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/pyarrow/_dataset.pyx:3226, in pyarrow._dataset.DatasetFactory.finish()

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowInvalid: Error creating dataset. Could not read schema from '/fh/working/sun_w/TRPCD/RP10_OCH-35__20260120__213109/proseg-output.zarr/points/transcripts/points.parquet/part.0.parquet'. Is this a 'parquet' file?: Could not open Parquet input source '/fh/working/sun_w/TRPCD/RP10_OCH-35__20260120__213109/proseg-output.zarr/points/transcripts/points.parquet/part.0.parquet': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

ArrowInvalid Traceback (most recent call last)
Cell In[1], line 35
33 cp.random.seed(seed)
34 adata_och35_10x = ad.read_h5ad("/fh/working/sun_w/sshen/TRPCD/OCH35_v1.h5ad")
---> 35 adata_och35_proseg = spatialdata.read_zarr("/fh/working/sun_w/TRPCD/RP10_OCH-35__20260120__213109/proseg-output.zarr")
36 adata_och35_proseg = adata_och35_proseg.tables['table']

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/spatialdata/_io/io_zarr.py:203, in read_zarr(store, selection, on_bad_files)
183 group_readers: dict[
184 Literal["images", "labels", "shapes", "points", "tables"],
185 tuple[
(...) 196 "tables": (_read_table, "tables", tables),
197 }
198 for group_name, (
199 read_func,
200 element_type,
201 element_container,
202 ) in group_readers.items():
--> 203 _read_zarr_group_spatialdata_element(
204 root_group=root_group,
205 root_store_path=root_store_path,
206 sdata_version=sdata_version,
207 selector=selector,
208 read_func=read_func,
209 group_name=group_name,
210 element_type=element_type,
211 element_container=element_container,
212 on_bad_files=on_bad_files,
213 )
215 # read attrs metadata
216 attrs = root_group.attrs.asdict()

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/spatialdata/_io/io_zarr.py:76, in _read_zarr_group_spatialdata_element(root_group, root_store_path, sdata_version, selector, read_func, group_name, element_type, element_container, on_bad_files)
70 element = read_func(
71 elem_group_path,
72 cast(Literal["image", "labels"], element_type),
73 reader_format,
74 )
75 elif element_type in ["shapes", "points", "tables"]:
---> 76 element = read_func(elem_group_path)
77 else:
78 raise ValueError(f"Unknown element type {element_type}")

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/spatialdata/_io/io_points.py:35, in _read_points(store)
32 path = store_root / f.path / "points.parquet"
33 # cache on remote file needed for parquet reader to work
34 # TODO: allow reading in the metadata without caching all the data
---> 35 points = read_parquet("simplecache::" + str(path) if str(path).startswith("http") else path)
36 assert isinstance(points, DaskDataFrame)
38 transformations = _get_transformations_from_ngff_dict(f.attrs.asdict()["coordinateTransformations"])

File ~/.conda/envs/GPU-SP/lib/python3.12/site-packages/dask/backends.py:151, in CreationDispatch.register_inplace..decorator..wrapper(*args, **kwargs)
149 raise e
150 else:
--> 151 raise exc from e

ArrowInvalid: An error occurred while calling the read_parquet method registered to the pandas backend.
Original Message: Error creating dataset. Could not read schema from '/fh/working/sun_w/TRPCD/RP10_OCH-35__20260120__213109/proseg-output.zarr/points/transcripts/points.parquet/part.0.parquet'. Is this a 'parquet' file?: Could not open Parquet input source '/fh/working/sun_w/TRPCD/RP10_OCH-35__20260120__213109/proseg-output.zarr/points/transcripts/points.parquet/part.0.parquet': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

[dcjones]

That's interesting. I've not encountered this issue or been able to reproduce it yet. I'd guess that proseg either ran out of memory or disk space when it was generating the output, but it's hard to be sure. Can you tell me how large that part.0.parquet output file is?

[sshen82]

That is 4.86gb. I am using the maestro server since i noticed you are also from the hutch. I am using 10 threads, but not monitoring the memory usage, so it could be running out of memory...