Using a different image size for linemod detection for KinectV2

[JimmyDaSilva]

Linemod in opencv is using a linearMemoryPyramid with levels {5,8} by default:
https://github.com/Itseez/opencv_contrib/blob/master/modules/rgbd/src/linemod.cpp#L1834:L1840

5 and 8 work fine for 640x480 images, but fails with every image sizes provided by the KinectV2 (512x424, 960x540, 1920x1080).
The problem has been discussed there:
opencv/opencv#4593

So I am trying to not use getDefaultLINEMOD but create my own detector using these lines in linemod_detect.cpp:

static const int T_LVLS[] = {3, 4};
std::vector< cv::Ptr<cv::linemod::Modality> > modalities;
modalities.push_back(new cv::linemod::ColorGradient());
modalities.push_back(new cv::linemod::DepthNormal());
detector_ = new cv::linemod::Detector(modalities, std::vector<int>(T_LVLS, T_LVLS +2));

instead of
detector_ = getDefaultLINEMOD()

For some image sizes the assertion then passes and the code runs. Unfortunately no objects are found...

@vrabaud I guess you know a bit more about the image processing behind. Do think this is something that actually work? Which values for T would you pick? How many pyramid levels do you think are necessary ?

Thanks for the help !
Jimmy

[nlyubova]

Hi,

I've done it once but it is not completely adapted for different images scales, check it in ecto_image_pipeline

nlyubova/ecto_image_pipeline@e855706

[nlyubova]

Regarding "For some image sizes the assertion then passes and the code runs. Unfortunately no objects are found..."

there can be another problem. You could try to train with the same image size that you will use for recognition (it worked for me), for example set in the training config file ;
renderer_width: 320
renderer_height: 240

[JimmyDaSilva]

I got some positive results yesterday. I will come back on this later. But thanks for the help ! :)

[nlyubova]

Did you try to change the threshold?

[nlyubova]

Setting images size in training config definitely helps, especially if training and test images are very different in size (that was twice in my case)

[JimmyDaSilva]

@nlyubova Don't worry more. It works great :)
You were absolutely right about adding the renderer_width and renderer_height in the training script. It changes everything!
I am now playing with the T_LVLS values in my code above for linemod_detect.cpp to get the best and fastest results.
I will post a new PR once I am done.

I didn't have to add the renderer_width and renderer_height for the detection script... so I don't really understand how this works.
But anyway it seems to work now !

[JimmyDaSilva]

Hooray ! Linemod on Kinect2 is coming for ORK :)

[nlyubova]

wow! good job! so what id you change?

[JimmyDaSilva]

In linemod_detect.cpp:

// detector_ = cv::linemod::getDefaultLINEMOD();
static const int T_LVLS[] = {4, 15};
std::vector< cv::Ptr<cv::linemod::Modality> > modalities;
modalities.push_back(new cv::linemod::ColorGradient());
modalities.push_back(new cv::linemod::DepthNormal());
detector_ = new cv::linemod::Detector(modalities, std::vector<int>(T_LVLS, T_LVLS +2));

And as you advised I have added in training.ork:

renderer_width: 960
renderer_height: 540

These params are for QHD images only.
For SD, I set T={2,4} renderer_width: 512 renderer_height: 424
Detection for HD takes for ever and never finishes.

[giacomodabisias]

Good Job.
I still believe that the code should be fixed in such a way that you can easily insert size and data type in order to use it with any depth camera. I will have a look soon and see if I can contribute.

[mikearmstrong800]

Thanks Jimmy and Nlyubova!

Thanks to you I get great linemod detection of the can with kinect2_bridge on ROS Kinetic on Ubuntu 16.04.

Mike