Can't load larger Image file

[SaibotMagd]

I try to load my example dataset (280GB) into mag using the roi parameter. But it won't load. I also tried to load it using the GUI, it say "loading image" but does nothing at all (I checked the CPU and w/r load for my system, the process is on hold). I've got 512 GB Ram in the system so this shouldn't be an issue (but mag using memmap anyway).
I will now cut it down and try again to find out the "maximal possible" filesize.

I already transformed it into the npy format:

magmap.io.cli - INFO - Set verbose to {<Verbosity.LEVEL: 1>: 1, <Verbosity.LOG_PATH: 2>: '/scratch/mouse_200928/stitched_image5d.npy'}
root - DEBUG - Trying logger path: /scratch/mouse_200928/stitched_image5d.npy
magmap.io.cli - INFO - Launched MagellanMapper from /home/toolboxes/magellanmapper/run.py
root - INFO - Set ROI offsets to [(30, 35, 30)], current offset (30, 35, 30) (x,y,z)
root - INFO - Set ROI sizes to [(25, 25, 12)], current size (25, 25, 12) (x,y,z)
root - INFO - Set maximum number of CPUs for multiprocessing tasks to 100
magmap.io.cli - DEBUG - Updating ROI profile for channel 0
magmap.settings.profiles - DEBUG - settings for 'default,lightsheet':
root - INFO - {'adapt_hist_lim': 0.1,
 'channel_colors': (<Cmaps.CMAP_GRBK_NAME: 'Green_black'>,
                    <Cmaps.CMAP_RDBK_NAME: 'Red_black'>),
 'clip_max': 0.5,
 'clip_min': 0,
 'clip_vmax': 98.5,
 'clip_vmin': 5,
 'colorbar': False,
 'denoise_size': 25,
 'detection_threshold': 0.1,
 'erosion_threshold': 0.3,
 'exclude_border': (1, 0, 0),
 'isotropic': (0.96, 1, 1),
 'isotropic_vis': (0.5, 1, 1),
 'load_rot90': 0,
 'max_sigma_factor': 2.8,
 'max_thresh_factor': 0.5,
 'min_sigma_factor': 2.6,
 'mp_max_tasks': None,
 'mp_start': 'fork',
 'norm': None,
 'num_sigma': 10,
 'overlap': 0.55,
 'points_3d_thresh': 0.7,
 'prune_tol_factor': (1, 0.9, 0.9),
 'resize_blobs': None,
 'scale_bar_color': 'w',
 'segment_size': 150,
 'settings_name': 'default,lightsheet',
 'sub_stack_max_pixels': (1200, 800, 800),
 'thresholding': None,
 'thresholding_size': -1,
 'tot_var_denoise': None,
 'unsharp_strength': 0.3,
 'verify_tol_factor': (3, 1.2, 1.2),
 'vis_3d': 'points'}
magmap.settings.profiles - DEBUG - None
magmap.io.cli - INFO - Set default (channel 0) ROI profile: default,lightsheet
magmap.io.cli - INFO - Set atlas profile to default
root - INFO - setting up Matplotlib style default
root - INFO - applying theme DEFAULT
magmap.io.cli - DEBUG - All command-line arguments: ['/home/toolboxes/magellanmapper/run.py', '--cpus', '100', '-v', '1', '/scratch/mouse_200928/stitched_image5d.npy', '--roi_profile', 'lightsheet', '--offset', '30,35,30', '--size', '25,25,12']
root - INFO - Loaded database from /home/.local/share/MagellanMapper/magmap.db
root - INFO - set multiprocessing start method to fork
root - INFO - No image filename set for processing files, skipping
root - ERROR - /home/anaconda3/envs/mag/lib/python3.6/site-packages/traits/observation/_has_traits_helpers.py:83: RuntimeWarning: Trait '_wrappers' (trait type: List) on class ActionItem is defined with comparison_mode=<ComparisonMode.equality: 2>. Mutations and extended traits cannot be observed if a new container compared equally to the old one is set. Redefine the trait with List(..., comparison_mode=<ComparisonMode.identity: 1>) to avoid this.
  RuntimeWarning,
apptools.preferences.preferences - DEBUG - loading preferences from </home/.enthought/mayavi_e3/preferences.ini>
apptools.preferences.preferences - DEBUG - loading preferences from <<_io.BufferedReader name='/home/anaconda3/envs/mag/lib/python3.6/site-packages/mayavi/preferences/preferences.ini'>>
apptools.preferences.preferences - DEBUG - loading preferences from <<_io.BufferedReader name='/home/anaconda3/envs/mag/lib/python3.6/site-packages/tvtk/plugins/scene/preferences.ini'>>
root - ERROR - /home/anaconda3/envs/mag/lib/python3.6/site-packages/traits/observation/_has_traits_helpers.py:83: RuntimeWarning: Trait 'adapters' (trait type: List) on class SegmentsArrayAdapter is defined with comparison_mode=<ComparisonMode.equality: 2>. Mutations and extended traits cannot be observed if a new container compared equally to the old one is set. Redefine the trait with List(..., comparison_mode=<ComparisonMode.identity: 1>) to avoid this.
  RuntimeWarning,
root - ERROR - /home/anaconda3/envs/mag/lib/python3.6/site-packages/traits/observation/_has_traits_helpers.py:83: RuntimeWarning: Trait 'adapters' (trait type: List) on class ProfilesArrayAdapter is defined with comparison_mode=<ComparisonMode.equality: 2>. Mutations and extended traits cannot be observed if a new container compared equally to the old one is set. Redefine the trait with List(..., comparison_mode=<ComparisonMode.identity: 1>) to avoid this.
  RuntimeWarning,
root - ERROR - /home/anaconda3/envs/mag/lib/python3.6/site-packages/traits/observation/_has_traits_helpers.py:83: RuntimeWarning: Trait 'adapters' (trait type: List) on class ImportFilesArrayAdapter is defined with comparison_mode=<ComparisonMode.equality: 2>. Mutations and extended traits cannot be observed if a new container compared equally to the old one is set. Redefine the trait with List(..., comparison_mode=<ComparisonMode.identity: 1>) to avoid this.
  RuntimeWarning,
root - INFO - set border to zeros
root - INFO - set border to [0. 0. 0.]
magmap.io.libmag - ERROR - No package metadata was found for magellanmapper
Traceback (most recent call last):
  File "/home/toolboxes/magellanmapper/magmap/io/libmag.py", line 1221, in get_version
    ver = metadata.version(config.APP_NAME.lower())
  File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 861, in version
    return distribution(distribution_name).version
  File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 834, in distribution
    return Distribution.from_name(distribution_name)
  File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/importlib_metadata/__init__.py", line 449, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: No package metadata was found for magellanmapper
root - INFO - setting up Matplotlib style seaborn
root - INFO - applying theme DEFAULT
root - INFO - got None selected
root - INFO - no roi found
root - INFO - Changed channel to [0]
mayavi.core.registry - DEBUG - Engine [<mayavi.core.engine.Engine object at 0x7eff41be30f8>] named Engine1 registered
root - ERROR - /home/anaconda3/envs/mag/lib/python3.6/site-packages/traits/observation/_has_traits_helpers.py:83: RuntimeWarning: Trait 'children' (trait type: List) on class KeyBindings is defined with comparison_mode=<ComparisonMode.equality: 2>. Mutations and extended traits cannot be observed if a new container compared equally to the old one is set. Redefine the trait with List(..., comparison_mode=<ComparisonMode.identity: 1>) to avoid this.
  RuntimeWarning,
root - ERROR - /home/anaconda3/envs/mag/lib/python3.6/site-packages/traits/observation/_has_traits_helpers.py:83: RuntimeWarning: Trait '_wrappers' (trait type: List) on class ActionItem is defined with comparison_mode=<ComparisonMode.equality: 2>. Mutations and extended traits cannot be observed if a new container compared equally to the old one is set. Redefine the trait with List(..., comparison_mode=<ComparisonMode.identity: 1>) to avoid this.
  RuntimeWarning,
root - INFO - camera: {'view': (0.0, 0.0, 1.0, array([0., 0., 0.])), 'roll': 0.0}
root - INFO - Scene activated with view: (0.0, 0.0, 1.0, array([0., 0., 0.])) roll: -175.0
root - INFO - Main image has not been loaded, cannot show ROI Editor
magmap.gui.vis_handler - DEBUG - TraitsUI preferences for magmap.gui.visualizer.Visualization: {'': (433, 297, 1200, 681), '$': <traitsui.key_bindings.KeyBindings object at 0x7eff28aa25c8>}

[SaibotMagd]

Interesting behavior. When I try to load a file it rename it:

Its a smaller one but still not working.

[yoda-vid]

Thanks for asking about this! Yeah memory shouldn't be an issue here because of memmap (and your computer's huge amount of RAM!). Are you trying to load the stitched.npy, 202.5 GB file, but then nothing loads? Was the stitched_image5d.npy text file created instead?

If stitched.npy was imported by another program, it may not work directly in mag because mag expects the NumPy array to be in t, z, y, x, [c] format and also looks for a metadata file (stitched_meta.yml). Does it work if you reimport the original file, which I'm assuming is stitched.tif? You could move aside the existing stitched_image5d.npy (to trigger a re-import) and run:

./run.py stitched.tif`

This should import it to stitched_image5d.npy and stitched_meta.yml. Then you can launch it by ./run.py stitched (which looks for the _image5d.npy file) or ./run.py stitched_image5d.npy. Let me know if that works.

[SaibotMagd]

I tried this but it doesn't work:

2021-12-16 10:41:36,272 - root - INFO - Looking for files for multi-channel images matching the format: /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro_ch_*.tif
2021-12-16 10:41:36,273 - root - INFO - Using the given single file /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro.tif
2021-12-16 10:41:36,284 - root - INFO - Extracting metadata for image import, may take awhile...
2021-12-16 10:41:36,286 - magmap.io.importer - INFO - Starting Java for Bioformats using JAVA_HOME set to: /home/anaconda3/envs/mag
2021-12-16 10:42:07,207 - root - INFO - tag: {http://www.openmicroscopy.org/Schemas/OME/2015-01}Image
2021-12-16 10:42:07,208 - root - INFO - tag: {http://www.openmicroscopy.org/Schemas/SA/2015-01}StructuredAnnotations
2021-12-16 10:42:07,208 - root - INFO - names: ['201205_Standard_mouse_4855_TOPRO-3_topro.tif']
2021-12-16 10:42:07,208 - root - INFO - sizes: [(2581, 1, 7578, 5735, 1)]
2021-12-16 10:42:07,208 - root - INFO - resolutions: [(1.0, -6979350000.0, -6979320000.0)]
2021-12-16 10:42:07,208 - root - INFO - zoom: 1, magnification: 1
2021-12-16 10:42:07,208 - root - INFO - pixel_type: uint16
2021-12-16 10:43:59,323 - root - INFO - filename: /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro
2021-12-16 10:43:59,324 - root - INFO - filename_base: /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro
2021-12-16 10:43:59,324 - root - INFO - Initializing multiplane image import planes to "/scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro_image5d.npy", may take awhile...
2021-12-16 10:43:59,326 - root - INFO - Loading file /scratch/mouse_201205/201205_Standard_mouse_4855_TOPRO-3_topro.tif for import
2021-12-16 10:43:59,327 - magmap.io.importer - INFO - Starting Java for Bioformats using JAVA_HOME set to: /home/anaconda3/envs/mag
2021-12-16 10:43:59,494 - root - INFO - loading planes from time 0, z 0, channel 0
2021-12-16 10:44:03,431 - root - INFO - setting image5d array for series 0 with shape: (1, 2581, 7578, 5735)
2021-12-16 10:44:04,242 - root - INFO - loading planes from time 0, z 1, channel 0
2021-12-16 10:44:04,262 - magmap.gui.import_threads - ERROR - Invalid Z index: 1/1
Traceback (most recent call last):
  File "/home/toolboxes/magellanmapper/magmap/gui/import_threads.py", line 85, in run
    fn_feedback=self.fn_feedback)
  File "/home/toolboxes/magellanmapper/magmap/io/importer.py", line 995, in import_multiplane_images
    series=series, rescale=False)
  File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/bioformats/formatreader.py", line 824, in read
    index = self.rdr.getIndex(z,c,t)
  File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/javabridge/jutil.py", line 961, in method
    result = call(self.o, name, sig, *args)
  File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/javabridge/jutil.py", line 892, in call
    result = fn(*nice_args)
  File "/home/anaconda3/envs/mag/lib/python3.6/site-packages/javabridge/jutil.py", line 859, in fn
    raise JavaException(x)
javabridge.jutil.JavaException: Invalid Z index: 1/1

But he still creates the image5d.npy file even if he can't load it afterwards (there's no .yml file). I think its an issue regarding to the z-t dimension. Sometimes, I can't find out when, imageJ saves the slices as timepoints so I have to retransform it, but I can't do this now because it costs to much time (at least 2 days), I try it again afterwards.

In the meantime I cutted a small 100vx³ block out of the dataset to check for the cell-detection in isolation. But the learning curve for the GUI and functions is so steep I can't handle it now, so I will do the cellcount using ClearMap2 as I did before. Time is running out and there's so much left to do.

[yoda-vid]

But he still creates the image5d.npy file even if he can't load it afterwards (there's no .yml file). I think its an issue regarding to the z-t dimension.

Thanks for posting this. Yeah I agree that it looks like something to do with the z- and t-axes. Is the shape of your image z = 2581, y = 7578, x = 5735? The image is being interpreted that way, though I am not sure why it triggers the invalid z index in Bio-Formats. Did your smaller block load ok?

But the learning curve for the GUI and functions is so steep I can't handle it now, so I will do the cellcount using ClearMap2 as I did before. Time is running out and there's so much left to do.

I completely understand. Makes good sense to use what you're familiar with. Always open to feedback on how to flatten the learning curve whenever you get the chance. Thanks for all your feedback thus far!

By the way, I am working on an update to load TIF files directly, without requiring import and also without requiring the whole image to be loaded. I hope this will help the setup/learning curve at least a little.

[SaibotMagd]

Thanks for posting this. Yeah I agree that it looks like something to do with the z- and t-axes. Is the shape of your image z = 2581, y = 7578, x = 5735? The image is being interpreted that way, though I am not sure why it triggers the invalid z index in Bio-Formats. Did your smaller block load ok?

Yes it did, also the detection seems ok, but I can't validate it because of GUI issues. Also for me all of the "settings" like "scale detection" (is says just 0 and I can't change it), ROI-editor: Filter, Bordered, Seg, Grid, MIP doesn't do anything. But the detection itself seems ok, but I can't validate them because I couldn't find out how to plot the detected cells onto the large top images (I can only see them in the smaller images below in the ROI-editor view). I don't know if this means anything but clearmap found 2400 Cells while mag only found 62, but its most likely a threshold issue.

By the way, I am working on an update to load TIF files directly, without requiring import and also without requiring the whole image to be loaded. I hope this will help the setup/learning curve at least a little.

That sounds like a good idea. For me, the most important challenge is the better handling of larger data sets. When I see that Imaris is able to display a dataset of over 500gb in 3D in less than 10s with their proprietary IMS format and scale it according to the zoom level, that's the benchmark (and the filesize is only 50% of the TIF). The same dataset can be exported by Imaris in 2 hours as Tif files, which takes ImageJ at least 4 days. I don't think the problem can be solved using Tifs, because as far as I know it is very difficult or impossible to load this format in parallel and piecewise (similar to memmap). Maybe you know napari and how they handle the problem (https://napari.org/tutorials/applications/dask.html)?

[yoda-vid]

Thanks for your feedback, as always!

Also for me all of the "settings" like "scale detection" (is says just 0 and I can't change it)

Yeah I realize this is confusing. The slider scales the sizes of blobs only in the 3D viewer and not in the ROI Editor.

ROI-editor: Filter, Bordered, Seg, Grid, MIP doesn't do anything

• Filter: when this is checked, running "Detect" should show the preprocessed image in the smaller ROI images
• Bordered: running "Redraw" should show a dashed rectangular border in the smaller images
• Seg: this is supposed to segment the blob in the smaller images after running "Detect", but I'm seeing a regression here on newer versions of Scikit-image and will need to fix this
• Grid: running "Redraw" should overlay black grid lines in the smaller images
• MIP: checking this should immediately change the overview images into maximum intensity projections across the ROI

I couldn't find out how to plot the detected cells onto the large top images (I can only see them in the smaller images below in the ROI-editor view)

Currently, the blobs are only shown in the smaller images. But good to know they could be useful to see in the overview images as well! Will note this as a possible future feature.

For me, the most important challenge is the better handling of larger data sets. When I see that Imaris is able to display a dataset of over 500gb in 3D in less than 10s with their proprietary IMS format

I agree that handling large files on commodity hardware is key. Imaris has definitely made admirable strides! I am trying a simpler approach by using the Tifffile library, which has memmap support for at least some types of TIFs. I've opened a PR (#90) to memory map these types of TIFs such as those output by ImageJ and BigStitcher to bypass the import step at least in these cases. It's not nearly as advanced as pyramidal images or using Dask (thanks for the link, by the way!) but can hopefully serve as a drop-in replacement for importing these files.