Improve handling of videos that are corrupt or have decoding failures

[mmokrejs]

Bug/Issue Description:

In bug #118 somebody else reported that dvr-scan silently stops processing its input. I have several HEVC containers manifestig same issue with dvr-scan v1.6 from pip. The HEVC file was created from many short 3gp files by ffmpeg.

dvr-scan --verbosity debug --min-event-length 2 --time-before-event 0.1s --time-post-event 0.1s --threshold 0.1 --kernel-size 3 --bg-subtractor MOG2 --output-mode opencv --add-region 5 816 843 984 2560 1492 2560 1920 1855 1920 1690 1551 0 1548 -i input.hevc -o output.mkv
[DVR-Scan] DVR-Scan 1.6
[DVR-Scan] Initializing scan context...
[DVR-Scan] Opened video input.hevc (2560 x 1920 at 25.000 FPS).
[DVR-Scan] Limiting detection to 1 region.
[DVR-Scan] Using subtractor MOG2 with kernel_size = 3
[DVR-Scan] Scanning input video for motion events...
Detected: 7773 | Progress: : 386991 frames [1:14:19, 88.38 frames/s][hevc @ 0x55e97abafd80] Duplicate POC in a sequence: 13.
Detected: 7775 | Progress: : 387014 frames [1:14:19, 86.78 frames/s]
[DVR-Scan] Processed 387014 frames read in 4459.8 secs (avg 86.8 FPS).
[DVR-Scan] Detected 7775 motion events in input.
[DVR-Scan] List of motion events:
-------------------------------------------------------------
|   Event #    |  Start Time  |   Duration   |   End Time   |
-------------------------------------------------------------
|  Event    1  |  00:00:07.4  |  00:00:00.2  |  00:00:07.6  |
...
|  Event 7775  |  04:18:00.4  |  00:00:00.2  |  04:18:00.6  |
-------------------------------------------------------------
[DVR-Scan] Comma-separated timecode values:
00:00:07.360,00:00:07.600,00:00:11.160,00:00:11.400,
...
,04:18:00.080,04:18:00.360,04:18:00.560
[DVR-Scan] Motion events written to disk.

I ran the same command now with --verbosity debug, there is little bit more info.

INFO: controller.run_dvr_scan(): Initializing scan context...
INFO: video_joiner._load_input_videos(): Opened video input.hevc (2560 x 1920 at 25.000 FPS).
INFO: scanner._handle_regions(): Limiting detection to 1 region.
DEBUG: detector.__init__(): Region Mask: area = (0,816),(2560,1920), coverage = 27.98%
INFO: scanner.scan(): Using subtractor MOG2 with kernel_size = 3
INFO: scanner.scan(): Scanning input video for motion events...
Detected: 7773 | Progress: : 386992 frames [1:56:06, 56.16 frames/s][hevc @ 0x55a89b671680] Duplicate POC in a sequence: 13.
Detected: 7774 | Progress: : 387009 frames [1:56:06, 54.36 frames/s]DEBUG: video_joiner.read(): No more input to process.
Detected: 7775 | Progress: : 387014 frames [1:56:06, 55.55 frames/s]
INFO: controller.run_dvr_scan(): Processed 387014 frames read in 6966.8 secs (avg 55.6 FPS).
INFO: controller.run_dvr_scan(): Detected 7775 motion events in input.
INFO: controller.run_dvr_scan(): List of motion events:
-------------------------------------------------------------
|   Event #    |  Start Time  |   Duration   |   End Time   |
-------------------------------------------------------------
|  Event    1  |  00:00:07.4  |  00:00:00.2  |  00:00:07.6  |
...

I tracked it back to the original 3gp file (51MB). mpv stalls in the problematic position with:

[ffmpeg/video] hevc: Duplicate POC in a sequence: 13.    
[ffmpeg/video] hevc: Error parsing NAL unit #0.
Error while decoding frame (hardware decoding)!

Maybe similar underlying issue is in iina/iina#3448 (no testcase)?

Meanwhile you can drop me an email so I can share some example files with you.

Would be nice if the List of motion events: table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.

Expected Behavior:
A clear error message if dvr-scan stopped reading the input, ideally reporting ffmpeg command to extract some data around the problematic position in the input stream (for copy&pasting).

Computing Environment:

• OS: Linux (Distro: Gentoo)
• Python Version: 3.11.8
• OpenCV Version: 4.9.0

$ ffmpeg 
ffmpeg version 4.4.4 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13 (Gentoo 13.2.1_p20240210 p14)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-4.4.4-r9/html --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --nm=x86_64-pc-linux-gnu-nm --strip=x86_64-pc-linux-gnu-strip --ranlib=x86_64-pc-linux-gnu-ranlib --pkg-config=x86_64-pc-linux-gnu-pkg-config --optflags='-O2 -pipe -march=native -ftree-vectorize' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-optimizations --disable-libcelt --enable-nonfree --enable-version3 --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-version3 --enable-version3 --enable-nonfree --enable-bzlib --enable-runtime-cpudetect --disable-debug --disable-gcrypt --enable-gnutls --enable-gmp --enable-gpl --disable-hardcoded-tables --enable-iconv --enable-libxml2 --enable-lzma --enable-network --enable-opencl --enable-openssl --enable-postproc --disable-libsmbclient --disable-ffplay --disable-sdl2 --enable-vaapi --disable-vdpau --enable-vulkan --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --enable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --enable-openal --enable-opengl --enable-libv4l2 --disable-libpulse --enable-libdrm --disable-libjack --enable-libopencore-amrwb --enable-libopencore-amrnb --enable-libcodec2 --enable-libdav1d --enable-libfdk-aac --enable-libopenjpeg --disable-libbluray --disable-libgme --disable-libgsm --disable-libaribb24 --disable-mmal --disable-libmodplug --enable-libopus --disable-libmfx --disable-libilbc --enable-librtmp --enable-libssh --disable-libspeex --enable-libsrt --enable-librsvg --enable-ffnvcodec --enable-libvorbis --enable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-cuda-llvm --disable-libflite --enable-frei0r --disable-libfribidi --enable-fontconfig --disable-libglslang --enable-ladspa --disable-libass --enable-libtesseract --disable-lv2 --enable-libfreetype --enable-libvidstab --disable-libvmaf --enable-librubberband --disable-libzmq --disable-libzimg --enable-libsoxr --enable-pthreads --disable-amf --enable-libvo-amrwbenc --disable-libkvazaar --enable-libaom --enable-libmp3lame --enable-libopenh264 --enable-librav1e --disable-libsnappy --enable-libsvtav1 --disable-libtheora --enable-libtwolame --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --disable-gnutls --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-vfpv3 --disable-armv8 --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-vsx --disable-power8 --disable-amd3dnow --disable-amd3dnowext --disable-fma4 --disable-xop --enable-pic --cpu=host --disable-doc --disable-htmlpages --enable-manpages
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

but also with:

$ ffmpeg 
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13 (Gentoo 13.2.1_p20240210 p14)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --docdir=/usr/share/doc/ffmpeg-6.1.1-r5/html --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --nm=x86_64-pc-linux-gnu-nm --strip=x86_64-pc-linux-gnu-strip --ranlib=x86_64-pc-linux-gnu-ranlib --pkg-config=x86_64-pc-linux-gnu-pkg-config --optflags='-O2 -pipe -march=native -ftree-vectorize' --disable-static --disable-libaribcaption --enable-avfilter --disable-stripping --disable-optimizations --disable-libcelt --enable-nonfree --enable-version3 --disable-indev=oss --disable-indev=jack --disable-outdev=oss --enable-version3 --enable-version3 --enable-nonfree --enable-bzlib --enable-runtime-cpudetect --disable-debug --disable-gcrypt --enable-gnutls --enable-gmp --enable-gpl --disable-hardcoded-tables --enable-iconv --enable-libxml2 --enable-lzma --enable-network --enable-opencl --enable-openssl --enable-postproc --disable-libsmbclient --disable-ffplay --disable-sdl2 --enable-vaapi --disable-vdpau --enable-vulkan --enable-xlib --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-zlib --enable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --enable-openal --enable-opengl --enable-libv4l2 --disable-libpulse --enable-libdrm --disable-libjack --enable-libopencore-amrwb --enable-libopencore-amrnb --enable-libcodec2 --enable-libdav1d --enable-libfdk-aac --enable-libopenjpeg --disable-libjxl --disable-libbluray --disable-libgme --disable-libgsm --disable-libaribb24 --disable-libmodplug --enable-libopus --disable-libvpl --disable-libilbc --enable-librtmp --enable-libssh --disable-libspeex --enable-libsrt --enable-librsvg --disable-ffnvcodec --enable-libvorbis --enable-libvpx --disable-libzvbi --disable-appkit --disable-libbs2b --disable-chromaprint --disable-cuda-llvm --disable-libflite --enable-fontconfig --enable-frei0r --disable-libfribidi --disable-libglslang --enable-ladspa --enable-lcms2 --disable-libass --disable-libplacebo --enable-libtesseract --disable-lv2 --enable-librubberband --disable-libshaderc --enable-libfreetype --enable-libharfbuzz --enable-libvidstab --enable-libvmaf --disable-libzmq --disable-libzimg --enable-libsoxr --enable-pthreads --disable-amf --enable-libvo-amrwbenc --disable-libkvazaar --enable-libaom --enable-libmp3lame --enable-libopenh264 --enable-librav1e --disable-libsnappy --enable-libsvtav1 --disable-libtheora --enable-libtwolame --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --disable-gnutls --enable-version3 --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-neon --disable-vfp --disable-vfpv3 --disable-armv8 --disable-dotprod --disable-i8mm --disable-mipsdsp --disable-mipsdspr2 --disable-mipsfpu --disable-altivec --disable-vsx --disable-power8 --disable-amd3dnow --disable-amd3dnowext --disable-fma4 --disable-xop --enable-pic --cpu=host --disable-doc --disable-htmlpages --enable-manpages
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

[Breakthrough]

If you are able, can you test running the video with another related project as suggested in the linked issue? It entails running PySceneDetect on the failing video with the following commands:

scenedetect -i video.mp4 --verbosity debug --backend opencv detect-content
scenedetect -i video.mp4 --verbosity debug --backend pyav detect-content

I would like to know if using a different library to handle video IO would resolve the situation, or at least be able to return an error that can be acted on. Will also try this once I have a sample, sent you an email just now. I might be missing some debug logging from PySceneDetect so will look into making sure that gets logged in DVR-Scan.

With OpenCV we try to read at least 5 more frames after a failure to decode (see https://github.com/Breakthrough/PySceneDetect/blob/main/scenedetect/backends/opencv.py#L282). Unfortunately there is no error that gets returned from OpenCV in the case of a decode failure.

Would be nice if the List of motion events: table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.

As long as you are not seeking in the input, you should be able to tell how many frames were processed from the log output:

[DVR-Scan] Processed 387014 frames read in 4459.8 secs (avg 86.8 FPS).

Thank you for the detailed report!

[mmokrejs]

Here I ran the command on just the problematic 51MB file (unlike merged HEVC file for whole day):

$ scenedetect -i input.3gp --verbosity debug --backend opencv detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
DEBUG: __init__.open_video(): Opening video with opencv...
DEBUG: context._open_video_stream(): Video opened using backend VideoStreamCv2
DEBUG: context.handle_options(): Initializing SceneManager.
DEBUG: __init__.detect_content_command(): Adding detector: ContentDetector({'weights': Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), 'kernel_size': None, 'luma_only': False, 'min_scene_len': 10, 'threshold': 27.0})
DEBUG: controller.run_scenedetect(): Running controller.
INFO: scene_manager.detect_scenes(): Downscale factor set to 10, effective resolution: 256 x 192
INFO: scene_manager.detect_scenes(): Detecting scenes...                                                                                                                                                               
  Detected: 0 | Progress:  37%|█████████████████████████████████████████████████████▍                                                                                           | 295/801 [00:02<00:04, 111.83frames/s][hevc @ 0x5641b3d58d80] Duplicate POC in a sequence: 13.
DEBUG: opencv.read(): Frame failed to decode.                                                                                                                                                                          
  Detected: 0 | Progress: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 800/801 [00:07<00:00, 103.77frames/s]
INFO: controller._detect(): Processed 800 frames in 7.7 seconds (average 103.76 FPS).
INFO: controller.run_scenedetect(): Detected 1 scenes, average shot length 46.9 seconds.
$ scenedetect -i input.3gp --verbosity debug --backend pyav detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
Usage: scenedetect [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Try 'scenedetect -h' for help.

Error: Invalid value for -b/--backend: Specified backend pyav is not available on this system!
$

Hmm, I have dvr-scan and scenedetect from pip. I have opencv on Gentoo system-wide.

OK, I installed https://pypi.org/project/av/ into same environemnt like dvr-scan and scenedetect.

$ pip install av
Collecting av
  Downloading av-12.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.6 kB)
Downloading av-12.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.7/34.7 MB 5.9 MB/s eta 0:00:00
Installing collected packages: av
Successfully installed av-12.0.0
$

Now I have:

$ scenedetect -i input.3gp --verbosity debug --backend pyav detect-content
INFO: context.handle_options(): PySceneDetect 0.6.3
DEBUG: context.handle_options(): User config file not found.
DEBUG: context.handle_options(): Parsing program options.
DEBUG: __init__.open_video(): Opening video with pyav...
DEBUG: pyav.__init__(): Restoring default ffmpeg log callbacks.
DEBUG: pyav.__init__(): Threading mode set: AUTO
DEBUG: context._open_video_stream(): Video opened using backend VideoStreamAv
DEBUG: context.handle_options(): Initializing SceneManager.
DEBUG: __init__.detect_content_command(): Adding detector: ContentDetector({'weights': Components(delta_hue=1.0, delta_sat=1.0, delta_lum=1.0, delta_edges=0.0), 'kernel_size': None, 'luma_only': False, 'min_scene_len': 15, 'threshold': 27.0})
DEBUG: controller.run_scenedetect(): Running controller.
INFO: scene_manager.detect_scenes(): Downscale factor set to 10, effective resolution: 256 x 192
INFO: scene_manager.detect_scenes(): Detecting scenes...                                                                                                                                                                                    
  Detected: 0 | Progress:   0%|                                                                                                                                                                                 | 0/801 [00:00<?, ?frames/s][swscaler @ 0x7f631cd76f80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:   0%|▏                                                                                                                                                                        | 1/801 [00:00<01:45,  7.56frames/s][swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdd2840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd87800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cd947c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:   1%|██▌                                                                                                                                                                     | 12/801 [00:00<00:13, 59.30frames/s][swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:   2%|███▉                                                                                                                                                                    | 19/801 [00:00<00:13, 59.98frames/s][swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdbb840] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdae880] deprecated pixel format used, make sure you did set range correctly

...

[swscaler @ 0x7f631ce4b380] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce93940] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:  37%|████████████████████████████████████████████████████████████▋                                                                                                         | 293/801 [00:02<00:04, 116.16frames/s][swscaler @ 0x7f631cdbd080] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cef3040] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce76ac0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cea1980] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce84b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ced4600] deprecated pixel format used, make sure you did set range correctly
[hevc @ 0x7f636ced2480] Duplicate POC in a sequence: 13.
[hevc @ 0x7f636ced2480] Error parsing NAL unit #0.
[swscaler @ 0x7f631ce84b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cee3b40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce0c700] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce68a80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce1a1c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
  Detected: 0 | Progress:  38%|███████████████████████████████████████████████████████████████▍                                                                                                      | 306/801 [00:02<00:04, 113.30frames/s][swscaler @ 0x7f631cebff00] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce4df80] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce935c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631cdede40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x7f631ce3f0c0] deprecated pixel format used, make sure you did set range correctly
CRITICAL: scene_manager._decode_thread(): Fatal error: Exception raised in decode thread.                                                                                                                                                   
  Detected: 0 | Progress:  39%|█████████████████████████████████████████████████████████████████                                                                                                     | 314/801 [00:02<00:04, 105.20frames/s]
Traceback (most recent call last):
  File "/home/mmokrejs/virtualenv/dvr-scan/bin/scenedetect", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/__main__.py", line 47, in main
    run_scenedetect(cli_ctx)
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/_cli/controller.py", line 58, in run_scenedetect
    scene_list, cut_list = _detect(context)
                           ^^^^^^^^^^^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/_cli/controller.py", line 101, in _detect
    num_frames = context.scene_manager.detect_scenes(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/scene_manager.py", line 895, in detect_scenes
    raise self._exception_info[1].with_traceback(self._exception_info[2])
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/scene_manager.py", line 918, in _decode_thread
    frame_im = video.read()
               ^^^^^^^^^^^^
  File "/home/mmokrejs/virtualenv/dvr-scan/lib/python3.11/site-packages/scenedetect/backends/pyav.py", line 279, in read
    self._frame = next(self._container.decode(video=0))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "av/container/input.pyx", line 208, in decode
  File "av/packet.pyx", line 80, in av.packet.Packet.decode
  File "av/video/stream.pyx", line 35, in av.video.stream.VideoStream.decode
  File "av/video/stream.pyx", line 44, in av.video.stream.VideoStream.decode
  File "av/codec/context.pyx", line 507, in av.codec.context.CodecContext.decode
  File "av/codec/context.pyx", line 404, in av.codec.context.CodecContext._send_packet_and_recv
  File "av/error.pyx", line 328, in av.error.err_check
av.error.InvalidDataError: [Errno 1094995529] Invalid data found when processing input
$

[mmokrejs]

Would be nice if the List of motion events: table contained also the frame starts and ends, so that one could see if the input was processed up to the very last frame or not.

As long as you are not seeking in the input, you should be able to tell how many frames were processed from the log output:

[DVR-Scan] Processed 387014 frames read in 4459.8 secs (avg 86.8 FPS).

So how many total frames were in the input file, I mean how many ffmpeg would find otherwise regardless of eventual failure)?

Or show column with percentages of the total input?

[Breakthrough]

So how many total frames were in the input file, I mean how many ffmpeg would find otherwise regardless of eventual failure)?

Or show column with percentages of the total input?

There's no easy way to show the expected total, some containers don't report it accurately and it is often incorrect. Using other libraries like PyAV allows ensuring that you have exhausted the full stream, but using OpenCV like this library currently does, it's not possible to do things like that. I think PyAV allows capturing those warnings, although I recall running into some deadlocks when doing that... I was recently made aware of some progress on the issue at PyAV-Org/PyAV#751 though so it probably is worth looking into to see if it's possible.

In the meantime this is similar enough to #118 and has been a huge help in the debugging process. I'll mark this as a duplicate of that, but feel free to continue the discussion there.

[mmokrejs]

Thank you for your efforts so far. I tend to believe the [ffmpeg/video] hevc: Error parsing NAL unit #0 is not always criticial and maybe the testcase cannt be processed due to the Error while decoding frame (hardware decoding)! instead. I see multiple [ffmpeg/video] hevc: Error parsing NAL unit #0 messages in a row while playng the movies.

I have several more testcases of mpv unplayable HEVC files (and causing dvr-scan to skip the remainder), but with different errors:

attempt 1
[ffmpeg] Failed to reallocate parser buffer to -2147482773
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] NULL: missing picture in access unit with size 579234816
(Buffering) V: 00:00:38 / unknown (71%) Dropped: 262

attempt 2
[ffmpeg] Failed to reallocate parser buffer to -2147483292
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] Failed to reallocate parser buffer to -2147483584
[ffmpeg] NULL: missing picture in access unit with size 1418189824
(Buffering) V: 00:00:24 / unknown (71%) Dropped: 129

https://trac.ffmpeg.org/ticket/8732
mpv-player/mpv#5893
mpv-player/mpv#5755

My CPU is Intel i-13900K chip so I use Intel QSV for hardware speedup (i915 kernel module).

My assumption is that all of this happens due to flakey ONVIF cameras and a switch dropping packets. Because of that, sometimes h264 video playing slows down or even stops, possibly while scanning for next full frame. Probably some combination of long series of events triggers a bug somewhere. It seems I will really need to re-encode the original 3gp files into different stream (h265) to ensure the incomplete frames are discarded. This also means the FPS (originally 15 or 25 fp) will appear variable frame rate (VFR) instead of fixed because many of the frames were randomly lost along the way to NVR. But then I will run into a different issue with dvr-scan as it does not support variable frame rate.

I think the design could be completely changed. A recording should be analyzed, regions of repeated movement (trees blowing in the wind) detected as glossy regions and their median color (a car may show up in front of them later so do not ask this region completely). In another sweep dvr-scan should look for motions in other places, even in a single frame. Further, dvr-scan detects some kernel size during it run. Provided I merge 3gp per 24-hrs into a single HEVC container (no re-encoding so far) all the videos start in night, in b/w color. I do not know what this kernel size is but certainly dvr-scan should split the video into ranges. This would be the very first pass of reading the file. Notably, the recordings at about the switch of IR cur filter are more grainy with distorted color and should be treated as a separate. Comparing individual/adjacent frames with each other does not make sense. The scene is largely static, sources of errors (small windblow movements can be learned) and then the real scene changes be detected more easily, even if recorded in a single frame.

Manual does not describe expectations so this is how I evolved my scripts to handle thousands of files per day and what I think is meaningful. Merging all videos per day into a single file is asking for a trouble but technically is the only way for me not to get lost in the zillions of files. It already sucks enough that ffmpeg -i list-of-files.txt ... -o merged.hevc dies due to the moov atom not found issue which means I have to edit the list-of-files.txt file and drop from it the problematic file and re-run ffmpeg.

[Breakthrough]

It seems I will really need to re-encode the original 3gp files into different stream (h265) to ensure the incomplete frames are discarded. This also means the FPS (originally 15 or 25 fp) will appear variable frame rate (VFR) instead of fixed because many of the frames were randomly lost along the way to NVR. But then I will run into a different issue with dvr-scan as it does not support variable frame rate.

The video output should be correct, as long as you are using the default OpenCV output mode. VFR video is only an issue for the accuracy of timestamps, which is why the ffmpeg ouptut mode won't work with it currently. That being said, there should be a filter for ffmpeg that will allow you to convert the VFR video into a fixed framerate one - you may want to consider that as a workaround.

A recording should be analyzed, regions of repeated movement (trees blowing in the wind) detected as glossy regions and their median color (a car may show up in front of them later so do not ask this region completely). In another sweep dvr-scan should look for motions in other places, even in a single frame

This sounds possible and like a good iea, but am lacking a lot of test videos which makes it difficult to develop these kinds of features. That being said, you may have some luck experimenting with the kernel size setting, which should help reduce small movements due to trees for example, and using the region editor to mask out problematic areas that aren't of interest.

[mmokrejs]

@Breakthrough Did you get my emails with links to the testcases causing dvr-scan to stop processing input files?

[Breakthrough]

@mmokrejs sorry for the delay, thank you for the samples I did receive them. I'll leave this open for now and re-word it as an investigation into improved logging on frame corruption. I haven't had any time to experiment with the videos you sent yet, but will let you know if I can come up with any improvements. In the meantime happy to accept any PRs if folks have similar fixes.

Many thanks for the test material.