too much polygons in one scene = ray tracing gives up in rendering

Development Rendering
, ,
1

Hi UDN!

I’m working on a short movie with lots of Metahumans in it.

We now have animated, displaced, and subdivided skeletal meshes on metahumans. There may be better ideas for performance, but we got it working with Nanite.

My 2 main characters don’t have nanite (we got some bugs [Content removed] but all the extra characters have Nanite working.

We came to the phase where we start lighting, and we notice a new problem: shadows aren’t cast anymore if raytracing is enabled.

I have two screenshots in the attachment.

The first one is with a rectangular light, without enabled raytracing.

The second one is with raytracing enabled, and for some reason without shadows.

Also, we only see this problem after rendering. We DON’T see this in play or editor mode.

The metahuman you see here is a displaced, subdivided metahuman with no nanite enabled.

The thing is, when muting all the extra characters in the background, all of a sudden, the shadows are cast again on our main character.

It looks like a memory problem…

So, we thought to work with light channels, and only throw the ray-traced shadows on our main characters, but that isn’t working.

Can we change some console variables? Or how can we avoid this?

We believe the problem is happening because we have too many polygons in one scene, the ray tracing gives up… but that doesn’t seem really the case as we don’t see this problem in editor mode.

Unfortunately, our project is 1.5 TB big, so it’s not a quick done to give you a test project…

Any ideas?

Whi is this happening while rendering and not in editor mode?

This is a project that started in UE4 so maybe we have some old render settings still in the game overrides option?

It looks like this:

[Image Removed]

Thank you!

2

The attachment seems to be lost in the previous post.

So here is with raytracing enabled:

[Image Removed]

and here without:

[Image Removed]

3

Hello there,

One thing that stands out as a potential to me is that the MovieRenderQueue (MRQ) configuration is has forced LODZero. If the editor is allowing the use of LODs, it’s possible that the MRQ is just pushing the polygon count too high if that is the issue. What is the primitive count in editor with stat unit?

Secondly, are you seeing anything in the logs that may be related to this?

As an aside, what GPU vendor is in use? There can be minor differences in how the drivers allocate the acceleration structures, but ultimately it is unlikely to be the issue.

Best regards,

Chris

4

Hi Chris!

Thanks!

I’m rendering on a NVIDIA GeForce RTX3090.

I couldn’t find anything in the logs that could explain this issue. I’ve attached a log of when I started up Unreal and rendered this specific shot.

I’ve rendered this without the use of LODZero.

This is a screenshot with stat unit on:

[Image Removed]

As you can see, I have some memory warnings, but most of the time I turn off displacement on all characters (I made a switch in blueprinting) and turn it on only when rendering.

In this screenshot, raytracing is on (and shadows are displaced correctly), and displacement is on.

Thank you!

5

Hi Chris!

Sorry for my late response. I was out of the office for some days.

I switched off the environment (because that had a lot of little lights), and removed and reduced all the lights to that one rectangular light with raytraced shadows enabled.

[Image Removed]Follow my chain of tests:

This is the result after rendering (displaced and subdivided metahumans):

[Image Removed]This is in editor mode I’ve disabled a lot of other metahumans (out of the frame), and now I don’t have that memory leak issue anymore.

[Image Removed]

The same in editor mode with stat unit and Stat D3D12RayTracing on:

[Image Removed]

This is the render with raytraced shadows disabled:

[Image Removed]

MegaLights are disabled in the project settings.

I’ve done a test with megalights enabled and enabled raytraced shadows, and this gives me after render this result:

[Image Removed]

The hair seems strange now, but the shadows seem better.

Thank you!

6

Hi Chris,

In addition to my previous reply, I stumbled upon the same issues with the male main character.

What I’m trying to add to my debug process is that ray tracing works when displacement is not on.

This is something you can follow in the first two screenshots:

[Image Removed][Image Removed]

So I tried to switch on displacement again:

[Image Removed][Image Removed]As extra info: one displaced (read: subdivided) metahuman has 607000 vertices and his body 779000 vertices

This white character doesn’t have nanite enabled, all the extra characters do (they are subdivided too).

I muted all the extra characters.

Displacement is still on, and I switched off megalights again.

And again, I have good results:

[Image Removed]

So, at least it’s consistent: I have good ray-traced shadows on one subdivided character (the rectangular light only shines on him by the way), but when I start to have more subdivided metahumans in the scene, ray tracing stops.

Thank you.

7

Hi Chris,

The main male character (BP_Arend) is the parent of all the others.

BP_Eva is the main female character. They are both white.

BP_Elsa and BP_Anke are extra characters. They are black.

When working in editor mode, I switch off the “displacement” boolean in the blueprint of Arend, as it is impossible to work.

When rendering, I switch it on.

This switch enables subdivided skeletal meshes on all metahumans, they use displaced textures.

When displacement is “on”, BP_Arend and BP_Eva don’t use nanite, as we got some bugs and disappearing triangles [Content removed]

But, all the extra characters do use nanite (they’re further from the camera, and triangles drop when characters are close).

To be able to use nanite on metahumans, we separated the face-skin from the eyes. Nanite doesn’t like the transparent materials of the eyes, so we haven’t displaced or subdivided the eyes and teeth.

I subdivided the face(skin) and the body using textures in the skeletal mesh tools. These were my settings:

[Image Removed]

I’ve managed to extract a test project, which can be found here:

I tried to simulate as many of the same conditions as I have in my project

In /All/Game/Cinematics-Tests you will find the sequence “Test_RayTracing_RectLight”

It can be slow to work if you try to open it when the “displace” boolean is on in BP_Arend (/All/Game/MetaHumans/Arend) - now it’s on.

I’ve rendered using the PreviewRenders_v01 preset, which is also available in the project. I’ve done a test without it, and it gives the same results.

But I noticed something new, I switched all extra characters off but forgot to switch off BP_Arend, and the render wasn’t okay, but not really the same as the previous render with all characters on… There seems to be an improvement when there are fewer polygons in the scene (see the mouth).

Renders can be found here: “RayTracing_RectLight\Renders\no shot\Comparison”

But for the completeness of this thread, I’ll add them here. The file names should explain what you see.

no shot_00001_RayTracingON_DisplacementOFF_AllExtraCharactersON+BP_ArendON

[Image Removed]

no shot_00001_RayTracingON_DisplacementON_AllExtraCharactersON+BP_ArendON

[Image Removed]

no shot_00001_RayTracingON_DisplacementON_AllExtraCharactersOFF+BP_ArendON

[Image Removed]

no shot_00001_RayTracingON_DisplacementON_AllExtraCharactersOFF+BP_ArendOFF

[Image Removed]

Thank you!

8

Hi Chris,

I’ve done a test with Pathtracer:

[Image Removed]

here you can find the results:

no shot_00001_RayTracingON_DisplacementON_AllExtraCharactersON+BP_ArendON_PathTracingON_FinalImage.00001.jpeg

[Image Removed]

no shot_00001_RayTracingON_DisplacementON_AllExtraCharactersON+BP_ArendON**_PathTracingON_PathTracer.00001.jpeg**

[Image Removed]

So the pathtracing doesn’t show the skin (which is subdivided and displaced)… and only two extra characters show up in the background.

I’ve done a test again with all extra characters and the male character off:

no shot_00001_RayTracingON_DisplacementON_AllExtraCharactersOFF+BP_ArendOFF_PathtracingON_FinalImage.00001

[Image Removed]

no shot_00001_RayTracingON_DisplacementON_AllExtraCharactersOFF+BP_ArendOFF_PathtracingON_PathTracer.00001.jpeg

[Image Removed]

So here the skin is pathtraced… Is there a limit to the path tracing?

Were you able to open the Dropbox link I’ve send in my previous comment?

Thank you!

9

I am able to see the dropbox, thank you very much for the repro.

I’ve searched for an documentation from the vendors on relevant limits, but there’s nothing that immediately forthcoming, so I’m going to pass this to somebody with more familiarity with the DXR.

One thing I did notice is that the body of the metahumans aren’t LODing at all, which will drive up the triangle counts on the hero characters. This may increase the number of BLASs that get created and, depending on what the limit is, may not change the result.

Best regards,

Chris

10

Hi Chris,

I appreciate your help. I’m looking forward to the follow-up to this thread.

I was just wondering where you see that the metahumans aren’t LODing?

I see every LOD activated in the skeletal meshes of the hero characters, or am I missing something?

All the best.

11

Hi Chris,

Sorry to check already. Do you have any news from somebody else about this problem?

All the best.

12

Hi Chris,

I’m hoping you’re doing well.

Is there a possible update on this thread?

Thank you!

13

Hi there, apologies for the delay. This looks like you might be running out of skin cache memory, but you should see a warning in your logs that looks like this:

OUT OF MEMORY FOR SKIN CACHE, REQUIRES XYZ extra MB (currently at XZY)

You can also view more information about the memory usage with the command r.SkinCache.PrintMemorySummary 1 which will print out the usage when out of memory, but you should try increasing r.SkinCache.SceneMemoryLimitInMB to see if that fixes the issue.

Hope this helps!

14

Hi Alex,

Thank you.

I didn’t find the error in the logs.

I’ve tried to increase r.SkinCache.SceneMemoryLimitInMB from 128 to 1024, and this didn’t help.

Afterward, I adjusted it to 10240, and it appears to have worked.

Are there consequences to setting this value that high? (How can I determine a good value)?

Thank you!

15

Hi Alex,

An Update. Got the r.SkinCache.PrintMemorySummary working**:**

[2025.06.17-13.46.34:713][178]LogSkinCache: Display: Used: 127.864MB

[2025.06.17-13.46.34:713][178]LogSkinCache: Display: Available: 0.136MB

[2025.06.17-13.46.34:713][178]LogSkinCache: Display: Total limit: 128.000MB

[2025.06.17-13.46.34:713][178]LogSkinCache: Display: Extra required: 1266.722MB

So that way, I can define a good value.

I’m still wondering if there are any consequences.

Thank you!

16

Are there consequences to setting this value that high? (How can I determine a good value)?

Using r.SkinCache.PrintMemorySummary is the best way to determine the amount of memory your scene requires. Increasing the r.SkinCache.SceneMemoryLimitInMB is more of a concern for real-time games where maintaining a memory budget is a critical. Because you are rendering out a sequence of frames you are only limited by hardware and time. Practically speaking, render highly complex scenes can take longer to render on the GPU which can cause TDRs, but you can often increase your TDRDelay to workaround that issue.

17

Super, thank you for the advice!

18

Interesting. I’ve tried a few things to replicate this on my end, such as creating a lot of instances of an animated skeletal mesh to push the primitive count up, forcing LOD0 on everything, and looking at a six million poly model without Nanite, but the issue isn’t replicating. How many RT shadow casting lights are in the scene?

The memory warning is a potential lead. Especially as exhausting the 3090 by 3Gi would suggest the scene is wanting 27Gi of GPU memory in editor, though that may not be indicative of the actual use when rendering as the editor can hold data around.

It also appears to me that it may be all lights that are incorrectly casting shadows. Would you be able to confirm what types of lights this may be affecting?

Additionally, are you using MegaLights?

Best regards,

Chris

19

Also, would you mind if I ask for the output of ‘Stat D3D12RayTracing’ as well?

That should give a bit more insight into the RT side of things.

Best regards,

Chris

20

Good findings.

This may be specific to dense skeletal meshes updating. I’d tested with a mixture of both and was still seeing accurate shadowing at over 44M polygons in the acceleration structures, but with notably fewer rebuilds.

Just to align my tests better, how are you subdividing and displacing the geometry?

Best regards,

Chris