At my university we have a virtual production studio with a setup comprised of a main wall, some side walls, driven by two render node PCs. Each render node is using Threadripper 7975WX along with nVidia RTX6000 Ada generation graphics cards. The main node has three RTX6000’s, and the second node has one.
I’ve been trying for weeks to successfully get a configuration working where we drive the outer frustum (split in two) on two of the GPUs, and the inner frustum on the third GPU. However, the documentation for multi-process is limited and as we have a generation of cards that does not support multi-gpu, we can’t use it.
In some configs I have managed to get the outer frustum on one GPU, the inner frustum on another, and the third remains idle, though in this case we still have synchronisation problems where the framerates of the two processes are not locked to our genlock at all, and there is often significant hitching in the framerate of the secondary “offscreen” process.
Is it at all possible to get in contact with someone from the @EpicGamesVirtualProd team to discuss and troubleshoot our setup?
Dont recommend multi GPU as Multiprocess is more efficient and you are going down that path anyway. Some of what you are describing splitting the frustum and rendering separately is more a 2110 config. You dont refer to being in a 2110 setup so it may ust be the terminology, which to be fair can get confusing. I would take a look at this link Converting a mGPU config to Multiprocess rendering You are not converting but the end result config is very similar to what you are trying to do. The final result has the wall split in two and the ceiling split in two and two wild wallls. Note the settings that need to go from SLI enabled to using mosaic in Multi process. If still stuck. Send a link to where I can download an empty project that includes your current nDisplay setup in a generic version of UE from the launcher and I will take a look.
Hi @ksc - sincerely sorry for the extreme delay in getting back to you. I’ve finally managed to secure some time in the studio to test and work with the setup this week.
I’ve updated to latest Unreal Engine, 5.5.4, and have been setting up again from scratch in case I missed something. I’ve used the ICVFX template as a baseline, and have the following config:
RN1 Huginn and RN2 Muninn are our render nodes. Each node is:
Threadripper 7975WX Pro
256BG RAM
RN1 Huginn has 3x RTX6000 Ada Gen cards, RN2 Muninn has 1x RTX6000 Ada Gen card.
Currently the setup is working okay - RN1 is rendering the outer frustum/main wall on one GPU, and the inner frustum on another using Multi Process. This seems to be working correctly. RN2 is rendering the movable side walls.
When running an environment (a Temples of Cambodia map from Fab), switchboard seems to indicate everything is operational:
Is it possible to render the left side of our wall, VP_0, and the right side of the wall, VP_1, on two different GPU’s, as well as the inner frustum on a third GPU, using Multi Process?
We are encountering significant hitching which appears to be the game thread of Node_2 (see below). Do you have any tips on where to look to troubleshoot this?
If, as in question 1, it’s not possible to utilize fully all three of our GPU’s in our primary render node, would you recommend moving one of them to the second render node and a different configuration in order to load balance the system properly?
After submitting this post I’ll zip and upload the project to somewhere you can download it and include here for further testing.
I’ve narrowed down the hitching I’m seeing as a long delay between the nodes acknowledging sync - the relevant timing event is “SharedMemoryMediaFrameAckTask”. I’m not sure what I can do to fix this? Any tips?
