Video length + Framing

[EmilBerheev]

Hey!
Was running your program with some tests on my local cluster and noticed following things:

• First stage of deciding on frames and resolutions script tend to fail on videos longer than ~20 minutes on following resources:
  • RAM 64GB, CPU: 16
  • The input was perfect podcast video with 3 usual camera perspectives
• Framing on the clips is static, so it only takes the frame space once and if speaker moves a little - it does not readjust to it, sometimes even on static recordings it tend to take speaker space wrongly

Are there are solutions to look for regarding those issues? I understand dynamic frame capture is not possible with current script basis, but maybe you do know what could be changed or added for it to work. Same regarding the video length

Attached a usual failure error on which script stops working:

2025-05-10 19:35:46,935 - DEBUG - Using 1 batches to extract and detect frames. Need 3.834 GiB of CPU memory per batch and 0.000 GiB of GPU memory per batch
2025-05-10 19:35:50,061 - ERROR - mmco: unref short failure
2025-05-10 19:36:27,752 - DEBUG - Detecting faces in 953 frames.
2025-05-10 19:37:34,757 - DEBUG - Detected faces in 953 frames.
2025-05-10 19:37:34,759 - DEBUG - Need 0.180 GiB to extract (at most) 70 frames
2025-05-10 19:37:34,759 - DEBUG - Face detection dimensions: 540x960
2025-05-10 19:37:34,760 - DEBUG - Need 0.101 GiB to detect faces from (at most) 70 frames

Thanks for amazing program!

[austinwmille]

Hey, thanks for being the first to test! Anyway, when mine failed on the 'detect faces' phase I assumed it was a memory issue. If you're running in Docker, is your total ram 64GB for the container? Anyway, here's my initial thoughts:

1. I believe the 'length' of the video is misleading us. You say your video is a great example, which I believe. I don't know why it would fail. In general, if you have a video that fails, try cutting the video into smaller segments instead of feeding the whole 20 minute video. This is crude, but it tends to work.

2. I don't know very much about how it does the framing. My gut feeling is that I doubt it will ever readjust the frame unless the speaker moves very far. As for the accuracy of the framing, now that IS optimizable, I think. Many of the packages are pinned to older versions. Some of the models are also pinned to smaller variations. I think it's using pyannote.diarization2.1.. I don't remember why it's not using 3.1. Anyway, I plan to experiment more with this once I have more free time.

Good luck, friend.

[austinwmille]

Was going thru issues and I realized that an addition to the script allowing it to utilize CUDA, if available, might help part of this issue.

I assume this script is a few years outdated too. If there is a better one you've found please post. Otherwise, if you could, please test a video again for me one month from now and tell me how it goes