Detected heap-buffer-overflow in src/quant.c:871:9

[Crispy-fried-chicken]

Hi, we have detected the heap-buffer-overflow in src/quant.c:871:9. The reproduce command is

-k -g -g -d jajuni -c 100x100+10+5 a_dither -h 60 -o -s average -e -f lum -d burkes /tmp/foo $PoC

and the asan trace stack is as follows:

�P0;0;1!z=================================================================
==431486==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000000d at pc 0x00000055a52b bp 0x7ffcde9bc4d0 sp 0x7ffcde9bc4c8
READ of size 1 at 0x60200000000d thread T0
    #0 0x55a52a in error_diffuse_normal /root/option/libsixel/src/quant.c:871:9
    #1 0x5549ab in diffuse_burkes /root/option/libsixel/src/quant.c:1074:9
    #2 0x5540b0 in sixel_quant_apply_palette /root/option/libsixel/src/quant.c:1529:25
    #3 0x4f378e in sixel_dither_apply_palette /root/option/libsixel/src/dither.c:800:14
    #4 0x4e1729 in sixel_encode_dither /root/option/libsixel/src/tosixel.c:826:27
    #5 0x4df963 in sixel_encode /root/option/libsixel/src/tosixel.c:1547:18
    #6 0x4dacc4 in sixel_encoder_output_with_macro /root/option/libsixel/src/encoder.c:1005:18
    #7 0x4d8902 in sixel_encoder_encode_frame /root/option/libsixel/src/encoder.c:1179:18
    #8 0x4d75b0 in load_image_callback /root/option/libsixel/src/encoder.c:1808:12
    #9 0x5025f9 in load_with_builtin /root/option/libsixel/src/loader.c:1010:14
    #10 0x501485 in sixel_helper_load_image_file /root/option/libsixel/src/loader.c:1651:18
    #11 0x4d72c7 in sixel_encoder_encode /root/option/libsixel/src/encoder.c:1872:14
    #12 0x4cd8b6 in main /root/option/libsixel/converters/img2sixel.c:464:22
    #13 0x7fde43390082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)
    #14 0x41c3cd in _start (/root/option/libsixel/build_asan/bin/img2sixel+0x41c3cd)

0x60200000000d is located 3 bytes to the left of 13-byte region [0x602000000010,0x60200000001d)
allocated by thread T0 here:
    #0 0x498b7d in __interceptor_malloc (/root/option/libsixel/build_asan/bin/img2sixel+0x498b7d)
    #1 0x4ce097 in rpl_malloc /root/option/libsixel/converters/malloc_stub.c:45:20
    #2 0x4dcd27 in sixel_allocator_malloc /root/option/libsixel/src/allocator.c:162:12
    #3 0x5401f5 in load_pnm /root/option/libsixel/src/frompnm.c:231:32
    #4 0x501d8e in load_with_builtin /root/option/libsixel/src/loader.c:890:18
    #5 0x501485 in sixel_helper_load_image_file /root/option/libsixel/src/loader.c:1651:18
    #6 0x4d72c7 in sixel_encoder_encode /root/option/libsixel/src/encoder.c:1872:14
    #7 0x4cd8b6 in main /root/option/libsixel/converters/img2sixel.c:464:22
    #8 0x7fde43390082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)

SUMMARY: AddressSanitizer: heap-buffer-overflow /root/option/libsixel/src/quant.c:871:9 in error_diffuse_normal
Shadow bytes around the buggy address:
  0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa[fa]00 05 fa fa 04 fa fa fa fa fa fa fa fa fa
  0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==431486==ABORTING

The PoC file is:

id:000008,sig:06,src:001969,time:12089493,execs:1171967,op:havoc_argv_comb,rep:8.zip

[saitoha]

@Crispy-fried-chicken Thank you for the report. For regression testing, may we add the PoC file you provided to the repository?

[Crispy-fried-chicken]

Of course, thank you for your consideration!

[saitoha]

@Crispy-fried-chicken I can’t reproduce.

The argument order in your example looks off:

-k -g -g -d jajuni -c 100x100+10+5 a_dither -h 60 -o -s average -e -f lum -d burkes /tmp/foo $PoC

• a_dither must be preceded by -d; as written, the parser treats a_dither as a positional input file.

• -o takes the next token as the output file, so this command would create a file literally named -s.

If you meant to set the dithering algorithm and then pass the other options, something like the following should parse correctly (examples):

# dither = a_dither
-k -g -g -d a_dither -c 100x100+10+5 -h 60 -s average -e -f lum -o /tmp/foo "$PoC"

# dither = burkes
-k -g -g -d burkes   -c 100x100+10+5 -h 60 -s average -e -f lum -o /tmp/foo "$PoC"

[saitoha]

cannot reproduce on master (v1.8.7); also not reproducible after checking out v1.8.6.

[saitoha]

at src/quant.c:871:9 on master (v1.8.6 and v1.8.7) I only see the harmless

    if (c < 0) {

so could you confirm which branch you tested

If it’s develop, there is

    c = *data + (error * numerator * 2 / denominator + 1) / 2;

which does involve a memory access; I suspected this and tried reordering the arguments, but I still couldn’t reproduce.

Could you share the exact branch/commit, full command line, and ./configure options?

[Crispy-fried-chicken]

Of course! I've tested in the master branch, and the tag is v1.8.7.
The ./configure options is

export CXX="clang++"
export CC="clang"
export CFLAGS="-fsanitize=address -g"
export CXXFLAGS="$CFLAGS"
./configure --prefix=`pwd`/gcov_install --disable-shared
make -j
make install

[saitoha]

@Crispy-fried-chicken
Is this problem still reproducible in your environment? I couldn't reproduce it on ubuntu-24.04, clang + libclang-rt-18-dev, -fsanitize=address -g, --disable-shared, so if you have any tips on how to reproduce it, please let me know.