Experiencing issues with data from the spinegeneric dataset

[NathanMolinier]

Hi, I am currently testing SpinenetV2 on the spinegeneric dataset and I am experiencing labeling issues and missed detections for T2w images, as displayed on the following images:

Could you explain me what is wrong with those images ? And how they are different from the training dataset used for SpinenetV2 ? Because according to your Readme.md T2w contrasts should be handled.

[rwindsor1]

Hi Nathan,

Thanks for the feedback. Indeed, T2 is the most represented sequence in the training set, so that shouldn't be a problem. I am surprised at this, as I have used the model on similar scans and not had any issue

Before we debug any further, could you confirm the following:

1. The pixel spacing is set correctly in the files. It may be worth changing it a bit to see if that affects quality. Have a look at the tutorial 2 to see how that is done.
2. Try setting scan_type to 'whole' when initialising spinenet, i.e. spnt = SpineNet(device='cuda:0', verbose=True, scan_type='whole'). This is a setting from a while ago which I should probably remove or change the default option of, but it might affect how the patch splitting is done.

Obviously if you can share the data that might help debugging this specific issue, although I appreciate that might not be possible.

Thanks,
Rhydian

[NathanMolinier]

The proposed changes seem to have improved the detection for some subjects, but the labels issue is still present for a lot subjects as shown here,

However, I'm not sure I understand correctly the spacing parameter: what is the correct unit ? Is it a resolution in mm/pixel or the inverse in pixel/mm? I tried both and got similar behavior.

Concerning the data that I'm using, it is in open access in the spinegeneric dataset. The last example I displayed is the T2w image of the subject sub-perform04.

The code I used is here,

code:

import matplotlib.pyplot as plt
from matplotlib.patches import Polygon
from spinalcordtoolbox.image import Image, get_dimension
from spinenet import SpineNet
import numpy as np

# load in spinenet
spnt = SpineNet(device='cuda:0', verbose=True, scan_type='whole')

# img_niftii --> 3D image: shape = (64, 320, 320)
img_niftii = Image(img_path).change_orientation("RPI")  # img_path is just a string with the path to the niftii image file
nx, ny, nz, nt, px, py, pz, pt = get_dimension(img_niftii)
nb_slice = 6 # Use less slices
img = np.rot90(np.moveaxis(img_niftii.data, 0, -1)[:, :, nx//2-nb_slice//2:nx//2+nb_slice//2])  # --> (320, 320, 6)

vert_dicts_niftii = spnt.detect_vb(img, 1/px)  # I also tried with px instead of 1/px

fig = plt.figure(figsize=(40,40))
for slice_idx in range(nb_slice):
    ax = fig.add_subplot(2,3,slice_idx+1)
    ax.imshow(img[:,:,slice_idx], cmap='gray')
    ax.set_title(f'Slice {slice_idx+1}', fontsize=60)
    ax.axis('off')
    for vert_dict in vert_dicts_niftii:
        if slice_idx in vert_dict['slice_nos']:
            poly_idx = int(vert_dict['slice_nos'].index(slice_idx))
            poly = np.array(vert_dict['polys'][poly_idx])
            ax.add_patch(Polygon(poly, ec='y',fc='none'))
            ax.text(np.mean(poly[:,0]), np.mean(poly[:,1]), vert_dict['predicted_label'],c='y', ha='center',va='center', fontsize=15)

fig.suptitle('Detected Vertebrae (all slices)', fontsize=100)
fig.savefig('visualize/test_spinenet.png')

[rwindsor1]

Hi Nathan,

Sorry for leaving this for a while, just looking into it now. I am not getting the issues with regards to labelling you are seeing but the detection is definitely not great so I am trying to investigate where the other vertebrae are going missing in the pipeline now. I have added a tutorial notebook tutorials/03-spinegeneric-example-detections.ipynb which shows my best effort. For sub-perform04, it seems to detect C3-C7 okay, but struggles with the thoracic spine in this image. I guess this is because the bottom of the image is very dark but I am investigating this further. I tried local contrast equalization but no luck so far.

I also tried it on sub_perform05 T2w and got very good results (T3-C3 all correctly detected), however no luck on sub_perform06 T2w.

To answer your question wrt. spacing, it should be mm/pixel.

Cheers, Rhydian

P.S. thank you for pointing out this dataset to us - we have not had much of a chance to test SpineNet's detection system on cervical scans, since we deal mainly with whole spine and lumbar FoVs. This should hopefully give us a chance to improve the system for cases like yours.

[NathanMolinier]

Thnks for your response ! By the way, I just tried with sub_perform05 and I got really bad results

Do you have any clue about what I am doing wrong ? Here is the part of my code where I'm using spinenet

import matplotlib.pyplot as plt
from matplotlib.patches import Polygon
from spinalcordtoolbox.image import Image, get_dimension
from spinenet import SpineNet
import numpy as np

# load in spinenet
spnt = SpineNet(device='cuda:0', verbose=True, scan_type='whole')

# img_niftii --> 3D image: shape = (64, 320, 320)
img_niftii = Image(img_path).change_orientation("RPI")  # img_path is just a string with the path to the niftii image file
nx, ny, nz, nt, px, py, pz, pt = get_dimension(img_niftii)
nb_slice = 6 # Use less slices
img = np.rot90(np.moveaxis(img_niftii.data, 0, -1)[:, :, nx//2-nb_slice//2:nx//2+nb_slice//2])  # --> (320, 320, 6)

vert_dicts_niftii = spnt.detect_vb(img, px)

fig = plt.figure(figsize=(40,40))
for slice_idx in range(nb_slice):
    ax = fig.add_subplot(2,3,slice_idx+1)
    ax.imshow(img[:,:,slice_idx], cmap='gray')
    ax.set_title(f'Slice {slice_idx+1}', fontsize=60)
    ax.axis('off')
    for vert_dict in vert_dicts_niftii:
        if slice_idx in vert_dict['slice_nos']:
            poly_idx = int(vert_dict['slice_nos'].index(slice_idx))
            poly = np.array(vert_dict['polys'][poly_idx])
            ax.add_patch(Polygon(poly, ec='y',fc='none'))
            ax.text(np.mean(poly[:,0]), np.mean(poly[:,1]), vert_dict['predicted_label'],c='y', ha='center',va='center', fontsize=15)

fig.suptitle('Detected Vertebrae (all slices)', fontsize=100)
fig.savefig('visualize/test_spinenet.png')

I'll compare my script with your tutorials/03-spinegeneric-example-detections.ipynb to figure out

[rwindsor1]

Hi Nathan,

I just tried running your script and I believe I have found the problem - in a nutshell, it comes down to the fact that subperform06 is isotropic, whereas we have used non-isotropic data while developing the model. The TLDR to fix the problem would be to make sure the entire vertebral volume appears in the image given to spinenet.

The pipeline works by extracting 3d volumes surrounding each vertebral body (see our technical paper, https://arxiv.org/pdf/2205.01683.pdf, figure 5 for example) and then the model makes initial predictions based on the appearance of those volumes. If you have non-isotropic slices (e.g. a pixel spacing of 0.8mm and a slice thickness of 3mm for example), you can see the edges of the vertebral bodies in the slices, especially at the top of the spine, and these slices give useful features for identifying the vertebrae. In the code you sent, the model can only see mid-sagittal slices which (because of the training data), it would assume are spaced a couple of mm apart from each other. Therefore the vertebra would appear much wider to the model so it would guess it was at the bottom of the spine, not the top.

You can fix this problem by sampling several slices spaced multiple mm apart. Here is an updated version of your code which produces the desired results. I have sampled 12 as opposed to 6 sagittal slices from the image and spaced them 0.8*4=3.2 mm apart.

import matplotlib.pyplot as plt
from matplotlib.patches import Polygon
from spinalcordtoolbox.image import Image, get_dimension
from spinenet import SpineNet
import numpy as np

img_path = 'example_scans/sub-perform05/anat/sub-perform05_T2w.nii.gz'
# load in spinenet
spnt = SpineNet(device='cuda:0', verbose=True, scan_type='whole')

# img_niftii --> 3D image: shape = (64, 320, 320)
img_niftii = Image(img_path).change_orientation("RPI")  # img_path is just a string with the path to the niftii image file
nx, ny, nz, nt, px, py, pz, pt = get_dimension(img_niftii)
print(px,py,pz)
nb_slice = 12
skip_slices = 4
img = np.rot90(np.moveaxis(img_niftii.data, 0, -1)[:, :, nx//2-(nb_slice*skip_slices)//2:nx//2+(skip_slices*nb_slice)//2:skip_slices])  # --> (320, 320, 6)

vert_dicts_niftii = spnt.detect_vb(img, 1/px)  # I also tried with px instead of 1/px

fig = plt.figure(figsize=(40,40))
for slice_idx in range(nb_slice):
   ax = fig.add_subplot(4,3,slice_idx+1)
   ax.imshow(img[:,:,slice_idx], cmap='gray')
   ax.set_title(f'Slice {slice_idx+1}', fontsize=60)
   ax.axis('off')
   for vert_dict in vert_dicts_niftii:
       if slice_idx in vert_dict['slice_nos']:
           poly_idx = int(vert_dict['slice_nos'].index(slice_idx))
           poly = np.array(vert_dict['polys'][poly_idx])
           ax.add_patch(Polygon(poly, ec='y',fc='none'))
           ax.text(np.mean(poly[:,0]), np.mean(poly[:,1]), vert_dict['predicted_label'],c='y', ha='center',va='center', fontsize=15)

fig.suptitle('Detected Vertebrae (all slices)', fontsize=100)
fig.savefig('visualize/test_spinenet.png')

Have a go with this and let me know if it works. Also, thanks for highlighting this - I'll make a hotfix which requires slice thickness to be given as input to detect_vb to stop other users getting confused by the problem!

[NathanMolinier]

Thanks for the fast response ! I tried to play with it and I got better results. However, in this script I was using 1/px for the spacing which is in pixel/mm and I believe it is not the right unit, we need to use px in mm/pixel.

• px = 0.8
• 1/px = 1.25

vert_dicts_niftii = spnt.detect_vb(img, 1/px)

Moreover, you proposed to take only 1/4 of the slices to have a non-isotropic input for the spinenet, but you kept the same spacing for the spinenet. Shouldn't we use the spacing multiplied by 4 as you mentioned here ?

spaced them 0.8*4=3.2 mm apart

[NathanMolinier]

I used this code and it seems better

import matplotlib.pyplot as plt
from matplotlib.patches import Polygon
from spinalcordtoolbox.image import Image, get_dimension
from spinenet import SpineNet
import numpy as np

img_path = 'example_scans/sub-perform05/anat/sub-perform05_T2w.nii.gz'
# load in spinenet
spnt = SpineNet(device='cuda:0', verbose=True, scan_type='whole')

# img_niftii --> 3D image: shape = (64, 320, 320)
img_niftii = Image(img_path).change_orientation("RPI")  # img_path is just a string with the path to the niftii image file
nx, ny, nz, nt, px, py, pz, pt = get_dimension(img_niftii)
pixel_spacing = np.array([py, pz])
nb_slice = 12 # Use less slices
skip_slices = 4
slice_thickness = px*skip_slices
img = np.moveaxis(img_niftii.data, 0, -1)[:, :, nx//2-(nb_slice*skip_slices)//2:nx//2+(skip_slices*nb_slice)//2:skip_slices]
scan = io.SpinalScan(img, pixel_spacing, slice_thickness)

vert_dicts_niftii = spnt.detect_vb(img, scan.pixel_spacing[0])

fig = plt.figure(figsize=(40,40))
for slice_idx in range(nb_slice):
   ax = fig.add_subplot(4,3,slice_idx+1)
   ax.imshow(img[:,:,slice_idx], cmap='gray')
   ax.set_title(f'Slice {slice_idx+1}', fontsize=60)
   ax.axis('off')
   for vert_dict in vert_dicts_niftii:
       if slice_idx in vert_dict['slice_nos']:
           poly_idx = int(vert_dict['slice_nos'].index(slice_idx))
           poly = np.array(vert_dict['polys'][poly_idx])
           ax.add_patch(Polygon(poly, ec='y',fc='none'))
           ax.text(np.mean(poly[:,0]), np.mean(poly[:,1]), vert_dict['predicted_label'],c='y', ha='center',va='center', fontsize=15)

fig.suptitle('Detected Vertebrae (all slices)', fontsize=100)
fig.savefig('visualize/test_spinenet.png')

[NathanMolinier]

Hi Rhydian,

This problem of labelling is more rare now, but still present for some subjects like sub-beijingGE01,sub-brnoUhb05 or sub-balgrist05... do you have the same issue ?

I tried with more slices here but the problem remains.

[NathanMolinier]

Hi Rhydian,

Did you try to use the T1w data in the spinegeneric dataset ?

[rwindsor1]

Hey Nathan,

Sorry for delays, been fairly busy the last two weeks with thesis submission & paper resubmissions. I did try with some T1w data but the model was doing poorly - I'm not sure the exact parameters of the sequence used but it visually appears different to some of the clinical images we've used before.

I want to investigate, however I'm away on holiday for the next week, so won't be able to look at it till the 29th.

[NathanMolinier]

Hi Rhydian,

Did you check how much different was your T1w training data ?

[rwindsor1]

Hi Nathan,

Again, sorry about the delay - I've just finished my PhD and started a new job so the time I can give to maintaining SpineNet is quite limited currently.

To answer your question - qualitatively, very different. I've attached below an image of some slices from a T1 sample in our training data and a T1 sample from your dataset (sub-perform05_T1w). As you can see, the intensity distribution is completely different

Example T1 scan from OWS (one of our training datasets)

Sub-perform05

I'm no expert on MRI protocols but my experience has been that in T1 scans we use the vertebral bodies are generally brighter than the discs and this seems to be true when I search 'T1 spinal MRI' online (e.g. https://case.edu/med/neurology/NR/MRI%20Basics.htm). Is there something atypical about the T1 protocol used here?

Either way, it looks like the images are sufficiently dissimilar to the training data that the model cannot pick up the vertebrae here. This is really interesting for us and we would like to explore this in more detail in the future and hopefully be robust to new, unseen sequences. However, as I stated, since I finished my PhD finding the time for me (or a new student working on this project) to do this will be tricky.

[NathanMolinier]

Hi Rhydian !

Thank you for the update, and no problem regarding the delay. It is really interesting how different the images are actually.

The protocol used for the spine-generic is presented in this paper, you can see the figure 9. Images were acquired from 3 vendors (GE, Philips and Siemens). See screenshot below:

Is it possible to have more information regarding the MRI sequence used for the OWS dataset ?