We have a crash with the Movie Render Queue, when we try to render 16K images.
At some Point the MRQ crash with the message GPU Out of Memory, but the Vram at the crash moment is not full.
[Image Removed]
We tested with different crads with 16 GB and 24 GB VRam, but always the same crash result.
Also we tested with 48GB VRam and the MRQ never crashed. But after investigate the VRam usage, we found that the Engine never consumes more that 15 GB of Vram. so it should work without any problem in a 24GB VRam configuration.
That’s why we believe the problem is caused by some buffer being filled, which depends on the total amount of VRAM.
Attached you can find the logs with the callstack and memory print after the crash.
How can we address the problem and avoid the crash?
We think what’s happening here is that you are making a request for an allocation of memory, and the GPU says, “No, I don’t have that amount available.” The engine doesn’t know what to do, so it crashes, which it probably shouldn’t. And it probably fails before it’s reported in the Dedicated GPU memory usage.
Is that screenshot with the 24gb card one where the render actually worked? We suspect you’re ultimately asking for a large allocation which fails before it’s reported in the Dedicated GPU memory usage. Whats the highest memory spike on look like on the 48GB VRAM machine?
In looking at the log you appear to be paying 2.5gb in SSR.TemporalAA (not sure if that’s TAA or Screen Space Reflections), which is kind of a lot if they’re not actually using those features. That + SceneColor are probably per-tile history, which is another 2.5gb. The guess is, because this is Movie Render Queue and not Graph, you might want to set “Render Warm Up Count” to 0, and change it to “Engine Warm Up” frames, and then check “Render Warm Up Frames”, and then also consider re-enabling texture streaming.
Ok so I talked with a handful of devs and product people. Nothing is jumping out to anyone where they can remember any memory leak or pool allocation issues like this. The closest thing I could find that may help is some work that went into 5.6 that revolved around Out of Memory issues where you can Page to System memory which downloads the per tile history.
You can probably seen whats happening by profiling the GPU while the MRQ render is running. You can hit the tilda key when the render window is up to get to a command prompt and run the profile.
Aside from that as mentioned before it’s a known issue in MRQ that having “Render Warm Up Frames” (instead of Engine Warm Up Frames) can cause a spike in vram usage that goes away after rendering starts so its possible that may be crashing the render before it can register to the details panel.
If you have debug symbols installed, the logs should provide some useful information.
Do you have texture streaming disabled in the Game Overrides section? If you have many high-resolution textures its possible this could be it. Many people don’t realize that when you disable texture streaming, UE will attempt to load all the mips for all the textures in the frame, plus some padding.
Ok, that’s good to hear! If you are rendering with the AA method set to none, it should not affect it. Rendering warmups are usually only necessary when GPU-based FX need to settle quickly. Or if there are big Lumen changes between shots.
Disableing “Allocate History Per Tile” seems to reduce the Memory by up to 2GB, but we need to test this more truly.
The Machine with 48GB VRam is not crashing but also only uses 16.66GB of dedicated GPU Memory at max. But the same Renderings are crashing on a machine with 24GBs of VRam. (Measured with GPU-Z)
Can you tell me if the Shared memory can be utilized fully when rendering with MRQ + HighResoltion (tiling) ?
And what would you suggest to use to identify what exactly is allocating the most GPU memory while doing these 16k renderings with MRQ ?
What is strange here is that we can render the exact same scene with a different camera angle without any issues, In that case it will consume a total of 14GB of VRam when rendering 16k.
And in some Camera angles it will crash with higher resolution. Rendering with 10k still works and consumes 15k of VRam but 12k crashes, my machine has 24GB of dedicated VRam.
So I highly doubt that this is because the GPU ran out of available total VRam.
We will try to profile it and get some more insights.
Are there any Command line parameter which are helping to get a useful callstack ? -d3ddebug and -gpucrashdebugging did not get us anywhere