Apply transformation matrix to images with double the resolution, but cannot overlap

[rockie-ding]

Hi @Gibbsdavidl,

Thanks for your wonderful program!

I currently try to align my HE image to MPM image (before registration, they differ in resolution already). However due to large file size, I downsize the resolution to get the registration transformation parameter first, then try to apply the found parameters to original image set (2X the resolution).

I used the get_slide and warp_img function, but after runnning the program, it seems the resulting warped HE and MPM images have different shapes. The error message
"scaling transformation for image with different shape. However, without knowing all of other image's shapes, the scaling may not be the same for all images, and so may not overlap."
pop up, which I think they cannot be registered given different shapes.

Do you know any de-bug to this?

`

Create a Valis object

registrar = registration.Valis(slide_src_dir, results_dst_dir)

Perform registration

rigid_registrar, non_rigid_registrar, error_df = registrar.register(reader_cls=slide_io.BioFormatsSlideReader)

==============================================================================

PART 1: WARP MPM

==============================================================================

slide_obj_mpm = registrar.get_slide('mpm_halfR')
warped_mpm = slide_obj_mpm.warp_img(
img="mpm_fullR.tif",
non_rigid=True,
interp_method='bicubic',
crop='overlap' # Use the standard overlap crop
)

Save MPM

cv2.imwrite(dst_f)

==============================================================================

PART 2: WARP H&E AND RESIZE TO MATCH MPM

==============================================================================

slide_obj_he = registrar.get_slide('he_halfR')
warped_he = slide_obj_he.warp_img(
img="he_fullR.tif",
non_rigid=True,
interp_method='bicubic',
crop='overlap'
)

Save H&E

cv2.imwrite(dst_f)`

[cdgatenbee]

Hi @rockie-ding,

This should be possible, potentially using a few different approaches that could achieve the same thing. But first, I wanted to ask if you are running into out-of-memory issues when you use the full size images. Valis should be able to handle the very large file sizes, since it doesn't actually read them into memory all at once. Valis will also resize them using libvips (fast and memory efficient) so the registration itself shouldn't be any slower or memory intensive. Or, is there a speed issue? Or something else?

Something I should note is that the Slide objects can actually warp images using two different functions. Slide.warp_img is intended for numpy arrays, so usually smaller images (like for thumbnails, images to be used for figures, etc...), while Slide.warp_slide uses libvips and is for warping larger images that wouldn't fit into memory if opened all at once. I realize this isn't all that clear, so I should probably update the examples and documentation to show the intended use case for each.

If you can use the full size image paths for registration, I'd recommend that, since it's a lot cleaner. Also, as you only have 2 images, and are aligning HE to MPM, you can make the reference image MPM and then you only have to save the warped HE image. That would look something like this:

from valis import registration, slide_io, slide_tools, warp_tools

# Perform registration, aligning to HE to MPM
ref_img = "mpm" # Will align H&E to MPM, so original MPM will be unwarped
registrar = registration.Valis(slide_src_dir, results_dst_dir, reference_img_f=ref_img)
rigid_registrar, non_rigid_registrar, error_df = registrar.register(reader_cls=slide_io.BioFormatsSlideReader)

# Only need to warp HE image since MPM is the reference image and didn't change
he_f = "he" 
he_slide = registrar.get_slide(he_f) # Get slide associated with the H&E image used for registration
he_slide.warp_and_save_slide("he_warped.ome.tiff)

However, if you're still running into issues with the large images (again, please let me know what these issues are so that I can work to fix them), I've put together some code that should hopefully work. Here, I'm aligning HE to MPM using the half resolution images, getting the full resolution HE image, warping it by scaling the transformation parameters (done inside of slide_tools.warp_slide), and then saving as a tiff (commented out, around line 36), or an ome.tiff that has the image's metadata but with updated information about size, resolution, etc...

from valis import registration, slide_io, slide_tools, warp_tools

ref_img = "mpm_halfR" # Will align H&E to MPM, so original MPM will be unwarped
registrar = registration.Valis(slide_src_dir, results_dst_dir, reference_img_f=ref_img)
rigid_registrar, non_rigid_registrar, error_df = registrar.register(reader_cls=slide_io.BioFormatsSlideReader)

full_he_f = "he_full" # Aligned H&E to MPM, so only need to warp H&E
he_slide = registrar.get_slide("he_half") # Get slide associated with the H&E image used for registration


# Need to know shape of full size reference image
full_mpm_f = "mpm_fullR"
full_mpm_reader = slide_io.BioFormatsSlideReader(full_mpm_f)
mpm_slide_shape_rc = full_mpm_reader.metadata.slide_dimensions[0][::-1] # originally W,H, needs to be row,col

slide_bbox_xywh, _ = he_slide.get_crop_xywh(crop=he_slide.crop, #Should be "reference"
                                        out_shape_rc=mpm_slide_shape_rc)

# Use above info to warp full resolution H&E slide
he_slide_reader = slide_io.BioFormatsSlideReader(full_he_f) # Note, if this is a .tif, try using slide_io.VipsSlideReader, which will be faster
full_warped_he = slide_tools.warp_slide(full_he_f, M=he_slide.M,
                                        transformation_src_shape_rc=he_slide.processed_img_shape_rc,
                                        transformation_dst_shape_rc=he_slide.reg_img_shape_rc,
                                        aligned_slide_shape_rc=mpm_slide_shape_rc,
                                        dxdy=he_slide.bk_dxdy, level=0, series=he_slide.series,
                                        interp_method="bicubic",
                                        bbox_xywh=slide_bbox_xywh,
                                        bg_color=he_slide.bg_color,
                                        reader=he_slide_reader)



# Save as a regular .tif without metadata. Uses libvips
# warp_tools.save_img("he_full_warped.tiff", full_warped_he)

# OR, save as ome.tiff with metadata
# Get OME metadata
pixel_physical_size_xyu = full_mpm_reader.scale_physical_size(0)

ome_xml_obj = slide_io.update_xml_for_new_img(img=full_warped_he,
                                                reader=he_slide_reader,
                                                level=0,
                                                channel_names=None,
                                                colormap=None,
                                                pixel_physical_size_xyu=pixel_physical_size_xyu)

ome_xml = ome_xml_obj.to_xml()

out_shape_wh = warp_tools.get_shape(full_warped_he)[0:2][::-1]
tile_wh = slide_io.get_tile_wh(reader=he_slide_reader,
                                level=0,
                                out_shape_wh=out_shape_wh)

# Save as ome.tiff
dst_f = "he_full_warped.ome.tiff"
slide_io.save_ome_tiff(full_warped_he, dst_f=dst_f, ome_xml=ome_xml,
                        tile_wh=tile_wh, compression="jpeg",
                        Q=100, pyramid=True)

I should note that this is based on the code inside of Slide.warp_and_save_slide, but I haven't put together an example dataset (create half resolution images, align using half resolution images, warp and save full resolution) to actually run it. So, you may run into some errors, but hopefully they're just typos. If you do, or run into any other issues, let me know and we can work through it.

Best,
-Chandler

[rockie-ding]

Hi @cdgatenbee , thanks for your fast reply, and I appreciate your help!

Unfortunatelty, when I try the first chunck of code with the original image, the error code like this always pop-up (this is also the reason why I want to reduce resolution first):
`Loaded LightGlue model
Loaded LightGlue model

==== Converting images

WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::load has been called by org.jpype.JPypeContext in an unnamed module (file:/C:/valis/.venv/Lib/site-packages/org.jpype.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
JVM has been initialized. Be sure to call registration.kill_jvm() or slide_io.kill_jvm() at the end of your script.
Converting images: 0%| | 0/2 [00:00<?, ?image/s]Converting slide to pyvips image
Converting images: 0%| | 0/2 [00:38<?, ?image/s]
C:\valis\valis\valtils.py:29: UserWarning: unable to call crop
extract_area: bad extract area

warnings.warn(warning_msg, warning_type)
Traceback (most recent call last):
Traceback (most recent call last):
File "C:\valis\valis\registration.py", line 4722, in register
self.convert_imgs(series=self.series, reader_cls=reader_cls, reader_dict=reader_dict)
File "C:\valis\valis\registration.py", line 2553, in convert_imgs
File "C:\valis\valis\registration.py", line 4722, in register
self.convert_imgs(series=self.series, reader_cls=reader_cls, reader_dict=reader_dict)
File "C:\valis\valis\registration.py", line 2553, in convert_imgs
File "C:\valis\valis\registration.py", line 2553, in convert_imgs
vips_img = reader.slide2vips(level=level)
vips_img = reader.slide2vips(level=level)
File "C:\valis\valis\slide_io.py", line 1322, in slide2vips
File "C:\valis\valis\slide_io.py", line 1322, in slide2vips
vips_slide = pyvips.Image.arrayjoin(
vips_slide = pyvips.Image.arrayjoin(
File "C:\valis.venv\lib\site-packages\pyvips\vimage.py", line 1324, in call_function
File "C:\valis.venv\lib\site-packages\pyvips\vimage.py", line 1324, in call_function
return pyvips.Operation.call(name, self, *args, **kwargs)
return pyvips.Operation.call(name, self, *args, **kwargs)
File "C:\valis.venv\lib\site-packages\pyvips\voperation.py", line 301, in call
raise Error(f'unable to call {operation_name}')
pyvips.error.Error: unable to call crop
extract_area: bad extract area

JVM has been killed. If this was due to an error, then a new Python session will need to be started`

When I try with the second chunck of code (half resolution first then full resolution), I got the below error msg:
`......
==== Measuring error

Measuring error: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2.76image/s]
Converting slide to pyvips image
QUEUEING TASKS | : 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 260/260 [00:02<00:00, 111.50tiles/s]
PROCESSING TASKS | : 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 260/260 [00:40<00:00, 6.40tiles/s]
COLLECTING RESULTS | : 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 260/260 [00:00<?, ?tiles/s]
Traceback (most recent call last):
File "C:\valis\registration_meAfterGatenbee.py", line 29, in
full_warped_he = slide_tools.warp_slide(full_he_f, M=he_slide.M,
File "C:\valis\valis\slide_tools.py", line 349, in warp_slide
vips_slide = reader.slide2vips(level=level, series=series)
File "C:\valis\valis\slide_io.py", line 1322, in slide2vips
vips_slide = pyvips.Image.arrayjoin(
File "C:\valis.venv\lib\site-packages\pyvips\vimage.py", line 1324, in call_function
return pyvips.Operation.call(name, self, *args, **kwargs)
File "C:\valis.venv\lib\site-packages\pyvips\voperation.py", line 301, in call
raise Error(f'unable to call {operation_name}')
pyvips.error.Error: unable to call crop
extract_area: bad extract area`

Do you know any fix to thie error?

[cdgatenbee]

Hi @rockie-ding,
This is certainly strange. It looks to me like there is some sort of error with reading the full resolution image. Do you know how the .tif was created? Can it be opened in QuPath? And how are you creating the half resolution version? I ask because sometimes .tifs aren't in a standard format that are easily read. For example, IndicaLabs .tifs are unique and can't (at least in the recent past) easily be read by BioFormats, libvips, or OpenSlide. So maybe the full resolution image is in a format that valis isn't reading correctly? There's probably a fix, but knowing how the full resolution tif and half resolution tif are being created might shed some light on this.

Best,
-Chandler