World Partition not taking Z space into account

Development World Creation
,
1

Hello fellow UE 5 trailblazers!

After days of targeted testing we acknowledge that Z space is not being taken into account by World Partition’s content loading. We have eliminated all other possibilities.

Imagine, if you will, a Blade Runner type city filled with content and much Z traversal. Now why isn’t something like that being considered?

We know that we are not using World partition as it was intended but we like to push limits and we have found the limit. We currently have close to 16,000 actors in a half finished level. Our video RAM sits around 4 to 5.5 Gigs. And our system RAM sits around 10-12 at its peak.

image

Upon entering areas with greater Z depth, rich with content, we experience lock up. Then after 2 minutes the render thread kills itself. This only happens when World Partition is enabled IN BUILDS, never in the editor. We do get well over 100 FPS in all areas. The lock up only happens in builds. Performance is smooth in the editor. The OS does not report the build as not responsive.

Admittedly our level is unique in that it is mesh based rather than terrain and the verticality could be called extreme when compared to traditional levels.

We understand that we could address this by using Streaming Volumes. We also understand that we could create smaller tighter levels. This would be working around World Partition’s limitations. Should we just accept that or is this something Epic intends to address?

OR do we need to write our own Z depth support?

Also curious if anyone else is trying to do something like us?
Please feel free to discuss. Thoughts and suggestions welcome.

Error for reference

Fatal error: [File:H:\current\UE5\Engine\Source\Runtime\RenderCore\Private\RenderingThread.cpp] [Line: 1243]
LogWindows: Error: GameThread timed out waiting for RenderThread after 120.00 secs

#if !WITH_EDITOR
#if !PLATFORM_IOS && !PLATFORM_MAC // @anonymous_user_5851a2121 MetalMRT: Timeout isn’t long enough…
// editor threads can block for quite a while…
if (!bDone && !bRenderThreadEnsured)
{
if (bOverdue && !bDisabled && !FPlatformMisc::IsDebuggerPresent())
{
UE_LOG(LogRendererCore, Fatal, TEXT(“GameThread timed out waiting for RenderThread after %.02f secs”), RenderThreadTimeoutClock.Seconds() - StartTime);
}
}
#endif
#endif

2 Likes
2

We figured out the problem a few days before 5.0 branch was updated and confirmed our problem. We were overwhelming the video card.

So we did a deep dive on performance and optimization starting with Virtual Textures. We VT’d as much as possible. This improved things but still not enough to run in 2560x1440 in Very High settings. Even in High or Medium settings we could still overwhelm the video card.

We looked at NVidia DLSS and AMD FidelityFX. We were a little disappointed in NVidia DLSS and it’s limitations. Not sure what the plan is there. However AMD FidelityFX WOW! Works cross platform, cross video cards and is open source.

And thanks to:

AMD FSR 1.0 for Unreal Engine 5 - (AMD FidelityFX Super Resolution) for porting it to UE5. Tested it on Mac OSX Big Sur and Windows. Definitly looking forward to 2.0 being released. This allowed us to reduced the secondary screen buffer and we could now run for awhile at 2560x1440 with

DefaultEngine.ini

[/Script/Engine.RendererSettings]

r.FidelityFX.FSR.Enabled=True

r.TemporalAA.Upsampling=True

r.TemporalAA.Algorithm=True

r.SecondaryScreenPercentage.GameViewport=70

r.TextureStreaming=True

r.VirtualTextures=True

r.VirtualTexturedLightmaps=False

r.VT.Residency.Notify=1

r.Streaming.DefragDynamicBounds=1

DefaultScalability.ini

[TextureQuality@3]

r.Streaming.MipBias=16

r.Streaming.AmortizeCPUToGPUCopy=1

r.Streaming.MaxNumTexturesToStreamPerFrame=1

r.Streaming.Boost=1

r.MaxAnisotropy=8

r.VT.MaxAnisotropy=8

r.Streaming.LimitPoolSizeToVRAM=1

r.Streaming.PoolSize=2500

r.Streaming.MaxEffectiveScreenSize=0

r.ScreenPercentage=77

Still not enough. So next step we created a DataLayer Loading Trigger Volume. Anything that is a prop in a room is put on it’s own data layer and when the player enters the trigger volume, we load and unload it at runtime. This seems to be working well. And lets textures get unloaded as nessessary.

We’re using 3 grids. Far, Mid and Near. Far is mostly architecture and will be used with HLODs. Current cell size is 6400 and loading range is 9600 with plans to change it to 6400 once HLODs are in. Mid cell size 3200 loading range 4800 and Near 1600 loading range 1600. We where using Near for props and such that we’ve moved to DataLayers now. So Near is used for decals, garbage on the ground, set dressing and stuff like that. So we plan to test it with a cell size of 3200.

With World Partition we found you need to have enough range between the loading ranges so you don’t try to load too many cells at the same time or you overwhelm the video card.

We will be introducing side loadable (instanced) levels so that we can unload the main streaming world. Seems to be the only way to clear the cache. We also had to drop back to DX11 as DX12 seems to overwhelm the video card faster and turned ray tracing off for now.

We had also forgotten about some work around we put in place during Early Access to combat some Lumen stuff that wasn’t finished. This included a secondary directional light (no shadows) and a post process volume that went over the entire level. Removing this got our WOW resolution and sharpness back up, solved our lighting (darkness) problem while bringing back the blackest of blacks. The directional light intensity during EA was set to 0.25, which is trash now. Setting it to the default intensity of 10 has made things look a lot better. And we need to do another lighting pass again, but we knew this going into to EA and being on the bleeding edge.

Video: UE 5.0.3 AMD FidelityFX TemporalAA.Upsampling - YouTube

Screenshot: ALARM_20220607.png - Google Drive

3

For anyone who’s interested, here’s what we’ve done so far.

4

Hey did you ever figure out why the level loading was slamming the system only in builds and not when you were testing from the editor?? I’m getting a similar issue with a LOT LESS than you guys have strewn about.

5

Hi JPMcWhiskers,

Yes! Ty for asking. We have figured out that the video card was being overwhelmed due to dense levels with so much verticality and content.

So, we have done multiple passes on our levels using Data Layers and Virtual Textures. We are now using Data Layers so that content only loads when you enter that Data Layer.

We have converted many textures to Virtual Textures and reduced the sizes of any textures that are on objects that are small on screen anyways.

It also might be worth noting that I am lighting the world for probably the 6th time now making sure that there are no overlapping light sources. We are getting good performance now. No more lock ups. Good frame rates.

This works well especially if you have a clean and well organized scene graph. All of our areas are organized neatly into descriptive folders so its really easy to find any of the 16000+ actors we have so far placed.

Using World Partition, Data Layer triggers and Virtual Textures is helping us create a complex game that will work well on multiple systems and video cards.

datalayers
datalayers656×857 65.4 KB

Edit:
Some tech info from partner…
Two other pieces… the Grid Sizes

Base - our simple base building blocks. Repeating textures, very simple objects with LOD

Far - Static objects that can be seen at a distant. DJV’s sign, roof, tree meshes … Again fairly simple objects

Med - More detailed complex architecture objects. Flags, statues, garbage cans. Scaffolding, graffiti

Near - ground Decals, damage decals on buildings, static mesh bushes, garbage

Our most recent config DefaultEngine.ini and DefaultScalability.ini here:

DefaultScalability.ini key areas changes are in the

[TextureQuality@] area

These 3 settings are all from low, because of the density of the world and the speed we can move through it while spiriting (probably won’t allow the player to move that fast), this helps to not overwhelm the video card

r.Streaming.MipBias=16

r.Streaming.AmortizeCPUToGPUCopy=1

r.Streaming.MaxNumTexturesToStreamPerFrame=1

DefaultEngine.ini key area

[TextureStreaming]

MemoryMargin=1000

This is due to our non streaming mipmaps. We were running at 900-1000megs… now we’re down to 700-800… Still working on it. (Virtualizing textures and reducing textures of small screen objects, like cups)

3 Likes
6

Hey hey heeeyyyyyy THANKS so much for the reply. These forums can be notoriously hard to get replies.

I started down the path this morning of using data layers and it looks like this is going to work really well. There’s also the possibility to create these data layers procedurally which is super helpful. I made a really quick and dirty volume layer activation blueprint and it seems to be the way to go.

Your reply is extremely reassuring. Thank you so much for it!!! Good luck on your project!!!

JP

1 Like
7

Hey, yvw! And TY.

Here’s a bit more info you might find helpful.
World Partition sizes…

worldpartition
worldpartition704×1212 143 KB

Small cell sizes works well for us because of the density of our level. Bigger cell sizes cause the video card to get overwhelmed. Video memory is stableizes at 4 gigs. Currently we are at about 60% texture pool used at very high settings. DefaultScalibility.ini [TextureQuality@3]

1 Like
8

That’s helpful to me because I was ignorant to what actually works, what people are trying, what can actually stabilize the spiking.

Supremely helpful, thank you so much!

1 Like
9

DataLayer use in UE5.1 update

Things have changed, and so have we. :slightly_smiling_face:

Unreal 5.1 Runtime DataLayers and World Partition

In Unreal 5.1 DataLayer behaviour has changed. Now we no longer need to unload layers. Before if you loaded a layer it would load the complete asset but just leave it hidden. Now if you load the DataLayer it appears to just load the meta data for the actors on the layer.

Once the layer becomes Active textures are now streamed. (assuming meshes too) This makes for smoother transitions in RunTime DataLayers.

UE5.1 Runtime DataLayers and World Partition | Unreal Engine Code Snippet

UE5.1 Runtime DataLayers and World Partition quick video runthrough

10

To address your initial concern.
“World Partition not taking Z space into account”

The World Composition - or whatever they renamed it in 5 (according to you World Partition)- has always been just QuadTree based.
As such, it has never supported vertical loading.
You’d need an octree system for that.

To achieve vertical loading, you have to create a custom system - probably as a component you shove into blueprints and use to load/unload blueprints, since its easier.

Unrelated as it may be: The way the landscape system kills performance, I’m not even sure there is any point waiting on Epic to make vertical loading a thing, since they will obviously base it on landscape stuff… (and to your performance issues. Once you have things finalized, consider converting from landscape to meshes, as you’ll probably gain 40% or so performance, particularly on older devices).

11

1 - The project has no landscape. It’s completely mesh based.
2 - We don’t wait. We find solutions and share them.
3 - It’s called World Partition.
4 - Our videos share our awesome performance with DataLayers and World Partition.

Your post has been flagged for being Off Topic as it has no relevance here.

12

Will that solution fits as well for a galaxy? The z-axis is pretty wide in this case.
Thanks in advance @Krucifear.

Also for an octree system, is there any teach of octree partitioning and / or applying lod on it? Or with an impotors system maybe?

13

Not aware of anything having made it into ue5 - you can always take c++ out of something like godot or any sample you may find and repurpose it to generate your own system.
Octree is a tried and true method that essentially guarantees performance when dealing with threedimentional instance management. In essence, its just a mathematical model of sorting things out so that computational overhead isn’t wasted on non needed calculations.

Though, thinking about it, they had to do something similar for Nanite to work, so maybe you can poke around the engine source and find classes already defined…

1 Like