Reduce near-zero-channel noise in multichannel gain maps

[gregbenz]

Summary

This reduces colored noise and edge artifacts in RGB gain maps, as shown in #487. The examples of improved results there use the approach from this PR. Both versions in those JPEG comparisons use full-resolution Q80 YCbCr 4:4:4 gain maps.

While the primary motivation for this PR is to improve image quality, it has a secondary benefit in that it reduces file size (likely caused by the artifacts not compressing as well). In a native test of 20 HDR/SDR pairs at roughly 2K and 4K in P3, all 40 files became smaller, saving 11.0% overall.

The approach impacts time to calculate the gain map modestly. Median API3 encoding time increased 3.5%, with individual configuration medians ranging from 1.3% faster to 6.1% slower. These timings were measured using a supplied, precompressed SDR base and exclude base creation and file I/O.

Change

When a channel is nearly zero in both SDR and HDR inputs, small input differences can produce large swings in gain. Those gains contribute little to that channel's reconstructed value, but lossy map encoding can spread their variation into other channels. This may result in significant noise or artifact, such as in areas of uniform color (such as blue sky gradients) or high frequency detail.

This PR uses a symmetric 203/255-nit offset in the gain calculation and matching normalized 1/255 SDR and HDR metadata offsets to make those ratios more stable. It builds on the offset approach in #358 while preserving the existing boost-range selection and caller controls.

The change applies to best-quality multichannel generation with supplied HDR/SDR inputs in the shared JPEG, AVIF and HEIF path. Single-channel, realtime and automatic HDR-to-SDR tone-mapping behavior remain unchanged. It does not change map subsampling and is independent of the AVIF/HEIF 4:4:4 proposal #486 (these PRs are analogous in terms of cross channel error causing artifact, but the underlying issues are different - with this PR relating to gain map calculation and the other PR addressing AVIF/HEIF encoding using chroma subsampling).

Tradeoffs

Quality is not uniformly better by every measurement. In the earlier 180-configuration size/gamut study, whole-image luminance error decreased in 161 configurations and increased in 19, with some local regions also showing higher error. The visual examples demonstrate useful improvements, not a guarantee for every image. The offset is an empirical compromise; smaller, larger and selective alternatives had mixed results.

Ranges that exclude 0 stops retain the original math to avoid clipping the adjusted gains. Explicit caller bounds excluding 0 bypass the new calculation up front. When such a range is discovered from the image, regeneration adds cost: an earlier synthetic test measured 56.3% longer API3 encoding. None of the 180 study configurations or 40 native timing configurations triggered this fallback.

Validation and measurement details

JPEG quality and size

• The earlier study covered 20 authored HDR/SDR pairs at three sizes and three gamuts, retaining the baseline boost ranges and display capacities. Settings were base Q90, full-resolution Q80 YCbCr 4:4:4 maps and gamma 1. All 180 containers became smaller, saving 12.77% in total. These numerical quality results are not a perceptual pass rate.
• The native implementation reproduced the study's maps and metadata in eight pilot cases. The full 180-case quality study was not rerun with the final native implementation; the later 40-case run measured native timing, metadata and sizes.
• A five-source calibration compared offsets of 1/510, 1/255 and 2/255, plus two selective alternatives. Halving the offset improved luminance RMSE on two charts but worsened it on three photos and increased aggregate file size by 1.29% relative to 1/255. This supports the chosen value for the tested inputs, not a universally optimal setting.
• The HDR JPEG crops were made without requantization or exposure adjustment. Inspect them in an HDR-capable viewer; displaying only the SDR base hides the gain-map differences.

Native timing

• All 20 pairs were tested at nominal 2K and 4K in P3 on Apple ARM64. Each configuration used one warmup and seven alternating mainline/candidate timing pairs. Sources below the requested size were not enlarged.
• Median gain-map generation time increased 6.5%. Median API3 time increased 3.5%; summing the per-configuration median times gave a 3.1% increase. All 40 native containers were smaller, saving 11.0% overall.
• The separate synthetic fallback test measured 84.1% longer gain-map generation and 56.3% longer API3 encoding on the preceding optimized implementation. The final gamma-one simplification does not affect that path, but the broader matrix did not retime it.

Correctness and compatibility

• All 67 gain-map math tests passed in both dual XMP+ISO and ISO-only builds. Coverage includes metadata offsets, caller bounds, natural one-sided ranges, gamma, default map scale and unchanged single-channel/realtime output. An asymmetric RGB fixture checks whole-map fallback when one ISO channel excludes 0 stops and coherent offsets when XMP's combined range includes 0.
• Earlier intrinsics-off validation passed 63 math tests, excluding three NEON-only tests and preceding the added asymmetric fixture. All 16 metadata tests and three focused JPEG tests also passed. The ISO-only encoded JPEG passed metadata parse-back with the baseline decoder.
• A bounded Apple ImageIO check recognized gain-map auxiliaries in six JPEGs and four AVIF/HEIF chart outputs. SDR and HDR decodes produced finite values. This checks reader recognition, not cross-platform pixel accuracy.

AVIF and HEIF

• Five inputs per codec were encoded as complete files through the existing 8-bit 4:2:0 map paths, at base Q90/map Q80, scale 1 and gamma 1. All ten pairs preserved decoded base pixels and weight-zero output exactly, with expected offsets and unchanged boost bounds, gamma and capacities.
• The three photos and chart produced smaller files in both codecs; fallback files were byte-identical. Full-HDR RGB error improved for the chart and noisy photo, changed little for the dark photo, and was essentially unchanged for the sky photo, including a roughly 0.01% AVIF increase.
• Three interleaved timing pairs per codec/input found photographic total encoding time near baseline. Synthetic fallback median overhead was 14.8% for AVIF and 6.2% for HEIF, with HEIF variation spanning no slowdown. This small screen does not establish performance neutrality or broad reader compatibility, and does not validate 4:2:0 as a sampling policy for gain maps.