Demoisaiced values are out of range

[Vincent-FA]

I'm passing an array of values [0, 65535] to vng4_demosaic and I'm getting values between -1632.537354 and 86103.359375 as a result. What's happening, what should I do?

With rcd_demosaic, I'm getting proper values.

[heckflosse]

We had this issue in RT as well. Last year I made some changes to RT demosaickers to avoid negative values. This changes are not yet in librtprocess. For now you could just check for negative values and clip them to zero. It's not only the case for vng4, also for amaze, where negative values occur sometimes at red/blue locations. I already searched the amaze code for the culprit though without success :(

[Vincent-FA]

What about the fact that values are also above 65535, should we clip them as well? A third above seems like losing a lot of information. Thanks

[heckflosse]

In RT we don't clip above 65535, only below zero. RCD implementation in librtprocess clamps to [0;65535]. Not sure what's the best. Though we should try to get consistent behaviour for the demosaickers.

[Vincent-FA]

What do you do with the values above if you don't clip them?

[darix]

@heckflosse which changes from RT need to be pulled into librtprocess?