Setting the "check_for_reflections option = True" prevents registration from running

[Mizuki-Kishikawa]

I attempted to perform registration between an HE image in .ndpi format and an mIF image in .ome.tiff format. When I first ran the process without enabling the check_for_reflections option, the program itself executed normally, but the registration failed because the source images were mirrored relative to each other.

When I set check_for_reflections = True, however, I encountered the following error message:

RuntimeError: expected m1 and m2 to have the same dtype, but got: double != float

I would like to know how to resolve this error.

The code executed is as follows:

import torch
import time
import os
from valis import registration

slide_src_dir = "src_dir" #Directory containing image files
results_dst_dir = "dst_dir" #Save location for results

# Create a Valis object and use it to register the slides in slide_src_dir
start = time.time()
registrar = registration.Valis(slide_src_dir, results_dst_dir, check_for_reflections = True)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()
stop = time.time()
elapsed = stop - start

The full text of the error message is as follows:

UserWarning: expected m1 and m2 to have the same dtype, but got: double != float
 warnings.warn(warning_msg, warning_type)
Traceback (most recent call last):
 File "/usr/local/src/valis/registration.py", line 4736, in register
  rigid_registrar = self.rigid_register()
           ^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/src/valis/registration.py", line 3401, in rigid_register
  rigid_registrar = serial_rigid.register_images(img_dir=self.processed_dir,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/src/valis/serial_rigid.py", line 1832, in register_images
  registrar.align_to_prev_check_reflections(transformer=transformer,
 File "/usr/local/src/valis/serial_rigid.py", line 1203, in align_to_prev_check_reflections
  matcher_obj.match_images(img1=reflected_img, desc1=reflected_desc, kp1_xy=reflected_src_xy,
 File "/usr/local/src/valis/feature_matcher.py", line 1457, in match_images
  match_distances, idxs = self.lg_matcher(t_desc1, t_desc2, lafs1, lafs2, hw1=hw1, hw2=r_hw2)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  return forward_call(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/kornia/feature/integrated.py", line 543, in forward
  pred = self.matcher(input_dict)
      ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  return forward_call(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/kornia/feature/lightglue.py", line 519, in forward
  return self._forward(data)
      ^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/kornia/feature/lightglue.py", line 589, in _forward
  encoding1 = self.posenc(kpts1)
        ^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  return forward_call(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/kornia/feature/lightglue.py", line 107, in forward
  projected = self.Wr(x)
        ^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
  return self._call_impl(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
  return forward_call(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/root/.local/lib/python3.12/site-packages/torch/nn/modules/linear.py", line 125, in forward
  return F.linear(input, self.weight, self.bias)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected m1 and m2 to have the same dtype, but got: double != float

[cdgatenbee]

Hi @Mizuki-Kishikawa,
Thanks for reporting this, and for sharing the full traceback error. It looks like what has happened is that features being matched aren't being converted to floats, which is what pytorch expects. This hadn't been an issue when the feature detectors expected numpy arrays, but the newer feature detectors expect pytorch tensors, which have to have the same datatype as the trained model. I have some example images I can use to debug this, and will let you know how it goes.

Best,
-Chandler

[Mizuki-Kishikawa]

Thank you for your prompt response.

[cdgatenbee]

Hi @Mizuki-Kishikawa,
I was able to update the code to avoid this error. I have a few other issues I need to address, but hope to push a more complete update in not too long. But, if you need the fix right away, and have installed valis so that it is editable, I can share the updated functions (feature_matcher.LightGlueMatcher.match_images and registration.Valis.extract_rigid_transforms_from_serial_rigid).

Best,
-Chandler

[Mizuki-Kishikawa]

Hi @cdgatenbee

Thank you for your prompt response.
Could you please share the updated function?

Best regards,
-Mizuki