Record with SceneCapture2D

[roushkaman]

Hello man! Immediately I warn you, I'm new to C ++. I want to rewrite your plugin a bit, if you do not mind that he would record with SceneCapture2D. Could you help me in this matter? Tf is the only one as long as who could make nvenc coding in the UE.

[ash3D]

Hi roushkaman!
The plugin uses VideoRecorder library internally. In general the following actions needed to accomplish recording: create CVideoRecorder object, call StartRecord/StartRecordNV, then call SampleFrame each time new frame rendered and finally call StopRecord. All this stuff implemented in VideoRecordGameViewportClient.cpp in the plugin. It can be reused in some extent to add support for SceneCapture2D. It is probably good idea to rearrange that code and extract common code to base classes/helper function to avoid implementing the same things twice.
There is also some functionality to implement for scene capture that may somewhat differs from current viewport implementation - grab frame data. For viewport UE allows to hook frame rendering - it takes place in UVideoRecordGameViewportClient::Draw. CVideoRecorder::SampleFrame called here. For scene capture different approach may be required. Good place for SampleFrame is probably UpdateSceneCaptureContents and CaptureScene.
As for actually reading back frame data the plugin currently employs 2 different approaches: generic ReadPixels for compatibility and optimized asynchronous readback which is implemented for D3D11. ReadPixels introduce severe performance penalty. The reasons is that it causes synchronization between game and rendering threads and also triggers CPU-GPU synchronization. Pipelining is key for good performance so UE's game thread queue up 1-2 frames for processing by render thread, it allows for multithreaded CPU processing. And DirectX generates command buffers for up to 3 frames ahead by default (the limit can be adjusted between 1 and 16 frames), it allow for parallel CPU/GPU work. ReadPixels breaks both these pipelining mechanisms and forces waitings between different CPU threads as well as between CPU and GPU. Thus I implemented asynchronous readback but I had not found required functionality in UE's RHI so I had to access D3D11 objects directly.
Recently I found alternative to ReadPixels in UE that exempts from dealing with underlying rendering API: Render Target Lookup (Non-blocking UpdateBuffer() section in the end). It avoids synchronization between game and render CPU threads. As for CPU-GPU synchronization, I looked at the UE sources and found out that it does not solve the problem - it passes 0 as map flags during mapping staging texture. Moreover it creates staging texture for each readback request (I used texture pool for D3D11 implementation). So presumably its performance should be somewhere between ReadPixels and D3D11 optimized asynchronous readback.

[roushkaman]

Thank you very much for the reply and explanation of your plugin! It remains a very important moment for me.
The point is that I would like to redo the implementation from the function UVideoRecordGameViewportClient::Draw in USceneCapture (Function render RenderTargetTexture). At this point, to intercept the current frame and give your library VideoRecord, but I don't know how to give a buffer, what would your library tech is understood and accepted. I previously mentioned that the C++ knowledge is weak and I would be grateful if you would please give this matter some attention.

In the future I will need to rewrite a little plug-in so that he would have the ability to record not in file on disk, and in the URL Stream. I understand that in the VideoRecorder the library will need to rewrite? Or you have implemented some configuration file?

Sorry for possible wrong text, I use a translator in Russian. =)

[ash3D]

VideoRecorder consumes frame data by calling CVideoRecorder::CFrame::GetFrameData function which plugin has to implement. But you do not have to bother about it unless you want to provide optimized implementation for rendering APIs other then D3D11 such as Vulkan or D3D12, but it would be quite challenging as it requires close interaction with UE4 in order to achieve high efficiency. All the needed buffer related functionality already implemented in UVideoRecordGameViewportClient::CFrame class and you can reuse it. This class has 2 specializations: one for ReadPixels based codepath and other for optimized D3D11 implementation. All you need to do is to provide corresponding UE4 object for which ReadPixels can be called for first specialization (replace viewport with this object in CFrame constructor) and provide D3D11 texture for second one (implement something similar to GetRendertargetTexture but for scene capture instead of viewport). I have looked through UE4 docs: USceneCaptureComponent2D has TextureTarget, you can try to call GetRenderTargetResource, it returns FTextureRenderTargetResource which has ReadPixels. It also has GetRenderTargetTexture function which returns FTexture2DRHIRef. Its GetNativeResource probably returns D3D11 texture object (if D3D11 renderer used of course). There are several other methods to retrieve FRHITexture, you can look at UE docs for details. It can turn out that some of them returns NULL so experiments needed. I had to employ 2 ways to get texture from viewport - one worked for editor and other for packaged game.
Take attention on which thread operation with UE object is permitted (game or render). It should be noted in UE documentation. If there are different alternatives, prefer render thread, it should avoid blocks and provide better performance. In order to perform some operation on render thread use one of the ENQUEUE_UNIQUE_RENDER_COMMAND_... macro - it places the op into queue which is processed by render thread. You can see its usage in UVideoRecordGameViewportClient::Draw as an example.
You can opt to implement only single readback mode (ReadPixels or native D3D11 texture) if you do not need other. For viewport plugin detects rendering API, then selects D3D11 implementation if possible and fallbacks to ReadPixels otherwise.
Hope my explanations are clear enough, ask for details on C++ otherwise.

As for networking functionality, I had not considered it when worked on the plugin. But eventually ffmpeg (which plugin is based on) provides it out-of-the-box. Try to pass URL as filename. Some minor modification for VideoRecorder can be required though.
First, additional format parameter (with values such as "flv" or "mpegts") can be needed. VideoRecorder currently extracts format from filename extension, and it will work for URL too if it contains extension. If not, format should be passed separately. I found ffmpeg based streamer on GitHub, you can see how to pass format parameter. VideoRecorder calls avformat_alloc_output_context2 here (note it pass NULL as format parameter which means it detects format based on filename, it is reasonable approach for files as it is very uncommon for files to not have descriptive extension).
Second, calls to avformat_network_init/avformat_network_deinit may be needed. ffmpeg's documentation states it is optional and can become mandatory at some version but it is unclear is it already the case.

Sorry for possible wrong text, I use a translator in Russian. =)

Russian is my native language, so feel free to communicate in it directly if it is more comfortable for you (however English can potentially be useful for others willing to comprehend the issue).

[roushkaman]

Thank you very much for this excursion! In general, your answer is very exhaustive! You are right, I am writing correspondence in this language specifically so that your help would be useful not only for me! =)
With network implementation, I have some experience, and if you want it can be used.
Once again, thank you very much, I will experiment and with luck I will necessarily share the results, and if not, I will continue to torment you if you do not mind! =)
With respect Michael!