Are there any plans to allow creating .avif and .heic images from base image + gainmap + cfg?

[MiikaH]

I was just trying out the new v2.0.1 to generate .avif and .heic multi-channel gain map ISO 21496-1 test images. Currently it seems like the ultrahdr_app CLI only supports avif/heic when using RAW intent HDR+base image pair, which after some fighting I was able to get working.

Eventually I was able to replicate my existing RGB gain map test image (shows in black and white in SDR and as full color image in HDR) on my browser HDR test set: https://miikah.fi/utils/browser_hdr_support

However I wasn't able to replicate exactly same image. AVIF version seems more saturated, HEIC looks less saturated (at least on Mac). I think the conversion method libheif uses, or my conversions from the jpg to the raw variants aren't quite able to generate identical gain map.

Not that big deal overall. The test pattern still works and the resulting images look good. But of course this made me wonder if there are plans to allow creating avif/heic images from plain base image + gain map itself, like for .jpg? So that exactly same workflow can be applied for all formats.

[MiikaH]

Hmm... When I try:
ultrahdr_app -m 0 -i color-base.jpg -g color-gainmap.jpg -f metadata.cfg -z rgb-gain-map-test-image-iso-21496-1.avif

I just get error: heif/avif encoding is supported only with raw intents

[MiikaH]

Yes indeed, looking at /lib/src/ultrahdr_api.cpp seems to confirm only raw input is currently supported.

@mohammadrezwankhan If above .avif and .heic commands work for you, you likely don't have UHDR_ENABLE_HEIF compile flag enabled, or don't have the ISO 21496-1 patched libheif dependency satisfied, so the build falls back to AVIF/HEIF disabled.

In latter case your commands are actually just producing .jpg files with wrong file extensions.

[MiikaH]

I did some more investigation to try and figure out why the colors are changing.

First, for reference this is best way that I have been able to come up to convert JPG gain map HDR image to AVIF/HEIC:

# HDR intent, linear half-float RGBA
ultrahdr_app -m 1 -j rgb-gain-map-test-image-iso-21496-1.jpg -o 0 -O 4 -z hdr_linear_f16.raw

# SDR base rendition, RGBA8888
ultrahdr_app -m 1 -j rgb-gain-map-test-image-iso-21496-1.jpg -o 3 -O 3 -z sdr_rgba8888.raw

# Generate the final image (-L 900 seems to best match the JPG)
ultrahdr_app -m 0 -p hdr_linear_f16.raw -a 4 -t 0 -C 1 -y sdr_rgba8888.raw -b 3 -c 1 -w 1024 -h 1024 -M 1 -s 1 -q 95 -Q 95 -D 1 -G 1.0 -L 900 -z rgb-gain-map-test-image-iso-21496-1.avif

Then I extracted the AVIF gain map with libavif avifgainmaputil extractgainmap to compare it to the original gain map. The gain map embedded in the AVIF file is much more faded out, so clearly all the parameters are different and there is less room for the color math to restore exact original RGB colors.

The original gain map + CFG file were generated by python script that inverts the gain map math, so that the output of grayscale base + RGB gain map + CFG settings equal mathematically exact match to the source color image.

That being said, they are close enough to be satisfying already. In both gain maps the HDR letters are each only visible in one color channel each (no luma or chroma bleed has happened between the channels), and the rendered HDR image is so close to the source that only side by side comparison makes the difference clear.

For reference the original JPG gain map source:

And the extracted AVIF gain map:

[DichenZhang1]

Thank you @MiikaH for trying it out and sharing the reproduction steps! A few observations:

Input Headroom is too small: Probing the input JPEG (ultrahdr_app -m 1 -P) shows --hdrCapacityMax 1.5 (~305 nits). This is too low for HDR, and the decoder clamps to 1.5 during linear reconstruction. Besides, the washed-out gain map isn't specific to AVIF/HEIC—re-encoding back to JPEG produces the same result.

I would recommend try using an HDR image with a higher hdrCapacityMax value. Let me know what you think. Thank you!

[MiikaH]

Yes, the file is intentionally generated with as little headroom as possible. Purpose for the test image is to pass as long as it is rendered as HDR, even if device/display has very little HDR headroom.

And indeed, the "washed out gain map" issue seems to come from the "to linear+sdr -> to gain map" back and forth chain. Therefore the original question was if it will become possible to feed the gain map image + CFG "directly", like it is with JPG. That leaves more room to craft pixel specific gain map images, where output is determined by manipulating the gain map formula (https://developer.android.com/media/platform/hdr-image-format#gain_map-generation), instead of expectation to display a natural HDR image.

[gregbenz]

@DichenZhang1 Is the min for HDR raised? I think even 204 nits worked before with v1 JPG (203 would output SDR without a map). There are images where 0.5 stops of headroom is useful (I wouldn't bother with much below that, but I think some artistic flexibility is ideal to allow encoding if that's what's requested in the source).

[MiikaH]

Huh, interesting. I tried to create a more "extreme" RGB Gain Map image to make the difference clearer:

Here is the original JPG. It shows different text and different emoji in the middle for both SDR and HDR rendition. There is slight fringing around sharp color changes:

Then I put it through the transforms required to make it .avif with a linear float16 HDR + SDR pair:

ultrahdr_app -m 1 -j rgb-gain-map-extreme.jpg -o 0 -O 4 -z hdr_linear_f16.raw
ultrahdr_app -m 1 -j rgb-gain-map-extreme.jpg -o 3 -O 3 -z sdr_rgba8888.raw
ultrahdr_app -m 0 -p hdr_linear_f16.raw -a 4 -t 0 -C 1 -y sdr_rgba8888.raw -b 3 -c 1 -w 512 -h 512 -M 1 -s 1 -q 100 -Q 100 -D 1 -G 1.0 -L 400 -z rgb-gain-map-extreme-transformed.avif

It becomes this:

It worked way better than I expected... Especially when using lower -L value. Now of course it still has a faint ghost image visible in the HDR version but that to me looks more like results of 4:2:0 chroma subsampling precision than the transform affecting it.

JPG version mostly avoided the ghost image because it uses the .jpg gain map that I feed it directly, which was compressed with 4:4:4 subsampling.

Regardless, it looks like the issue is about lack of exact control with the current AVIF/HEIC workflow. But if results don't have to be pixel precise, then even the current workflow is fine.

EDIT: With upscaled image and adjusted background even the current AVIF workflow can mostly eliminate the artifacts. Although iOS seems to do some kind of interpolation on the gain map that makes this kind of "extreme" test images problematic