So I’ve been working on a rather large personal project as of late, and I’ve been attempting to set up SWARM on my home server, which is a decently powerful machine and, I got to thinking. Why not also allow shader compiling over the network? So let’s say a moderately sized team is working on a large game project, and during asset implementation phase at that. It would be very useful to have a UE4 build tool server running on a powerful computer to task and handle all shader compiling. I realize this would require a shared DDC, which is fine as most teams of this size would be using that anyway. If the compiler machine is reasonably more powerful than the editors workstation, this could drastically improve efficiency on the end of the editor. Furthermore, if a build server were set up, reflection captures, geometry and third party integrations such as Steam Audio could be built on this server machine as well, further freeing up resources on the editors machine.
I’ve run some tests using the tools available to test my theory, and while it was very tricky to pull off it did indeed prove my point to some extent. I set up two machines:
Machine 1:
Intel Core i9-9900X
16 GB RAM
RTX 2080
Machine 2:
Intel Core i9-9900X
80 GB RAM
2x Titan RTX
Machine 1 is set up as the editing workstation, and Machine 2 is the server to be handling compilation/building.
The test project was being stored on a NAS and accessed on both machines via a SMB mount; both are set up using a shared DDC which is stored on the same NAS the project is stored on.
The test goes as follows:
Machine 1 imports some textures and creates a material, then Machine 2 should compile said shaders when the apply or save button is hit or during material creation. Now I am not able to do this with the current version of UE4 so I created a sort of macro application on Machine 2 in visual studio that automatically creates a material when it detects the base color texture in the folder its looking at, adds the normal, roughness, sets the material to translucent (heaviest on shader compilation in my experience) and hits save. I timed the moments the macro was doing something and the time it took to compile the shader for 4 materials and created some materials on Machine 1 as a control.
Note that each time Machine 2 saves a material and it’s finished, it takes a few seconds to show up in the File Browser on machine 1, this may seem insignificant but adds 1 to 3 seconds per material saved with remote compilation
ALSO – after Machine 2 finishes compiling and saving materials, it can vary wildly on how long Machine 1 took to load the shaders and display the materials properly. from 5 seconds to 50 seconds per material saved with remote compilation. I am leaving these two times out of the results below as we are talking about freeing up resources and improving efficiency. The editor having to wait a few extra seconds for the mats to show properly is way better than waiting for them to compile locally and not being able to continue working during compilation.
Scenario 1
I imported 9 4K textures on machine one and Machine 2 ran it’s macro. All the while I was able to continue importing textures and doing work while Machine 2 did it’s thing. Now the feature I’m requesting the editor would still need to make the material, so I accounted for roughly a minute and a half that it would take me to create such a material and subtracted it from the result time.
[SPOILER]
S1 Results:
Importing textures -- **1 min 52 sec**
** Time it would take editor to create material: **1 min 22 sec**
shader compilation time on Machine 1: **N/A**
--Shader compilation times per material on Machine 2--
Mat 1: **1 min 20 s**
Mat 2: **1 min 15 s**
Mat 3: **1 min 16 s**
Mat 4: **57 s**
[/SPOILER]
Total time for editor user before being able to move on to more work: 3 minutes 23 seconds
*** Note this is not counting any shader compilation time, just the time it took to make all the necessary clicks and wait for the editor to catch up; shader compile time is timed separately or during any time the shader compiler process is running and when UE4 says “Compiling Shaders”*
Scenario 2
I imported 9 4K textures on machine one and created the materials myself, allowing the shaders to compile on my editing workstation.
[SPOILER]
S2 Results:
Importing textures -- **1 min 59 sec**
** Time took to create materials: **3 min 14 sec**
Shader compilation time on Machine 2: **N/A**
-- Shader compilation times per material on Machine 1 --
Mat 1: **1 min 30 s**
Mat 2: **1 min 28 s**
Mat 3: **1 min 28 s**
Mat 4: **1 min 26 s**
[/SPOILER]
Total time for editor user before being able to move on to more work: 11 minutes 8 sec
Now, as the requested feature would obviously not be automatically creating materials, let’s replace the time it took to create materials in Scenario 1 with Scenario 2s time it took to create materials. This will tell us roughly how long it would take an average editor to import some textures, create a few materials and save said materials before being able to move on to more work.
RESULTS:
[SPOILER]Here we will see a very, VERY, VERY rough estimation of how long it would take to do the actions stated above but have a second machine handling the heavy lifting (compiling shaders)
Total time to import textures, create materials and move on while Machine 2 compiles shaders: 5 min 1 sec
VS
Total time to import textures, create materials and WAIT while current machine compiles shaders: 11 min 8 sec[/SPOILER]
You can see the stark difference there. Again, this is a super rough estimation, and I didn’t have much time to do more testing and validate my results, but I have a pretty good feeling they would be near the same. I’m also still rather fuzzy on how the shared DDC system actually functions as a whole due to the lackluster documentation in that area. It’s not hard to set up, but understanding it is a bit fuzzy. Does it share everything from the local DDC? But I’m getting off track.
Now this test has been primarily focused on shader compilation during material creation (when hitting apply or save) – this will also significantly boost editor performance though when the editor is compiling shaders in the background, such as when migrating a large amount of assets from one project to another or while editing the actual material itself. Shader compiling can hog up resources, so during this process it can make the editor almost unusable. Having it offloaded to another machine will free up all of these resources.
Anyway, I hope you guys think about this as I do believe it would increase editor efficiency among development teams.
PS: Does anyone know why sometimes compiling shaders sometimes goes into the negative? For instance it’ll start saying “Compiling Shaders 500” then count down to “Compiling Shaders -201”