We are using LevelStreamingVolumes. In an elevator setup, we load the upper floor while going up the elevator. Then we open the elevator doors based on a timer. The LSV for the upper floor is right above the first floor, so the player hits it right after going up. The LSV is set to loading and visibility. The doors open based on a timer, not based on the sub-level loading status.
What happens soon after we hit the LSV for the upper floor, the game thread has large frametimes as it waits for the VirtualTextureSystem to load. The game thread will be stuck in STAT_GameIdleTime for ~420ms over multiple frames. Each frame waiting on FVirtualTextureSystem::GatherAndSubmitRequests.
Our texture pools do not look over subscribed using r.VT.Residency.Show.
I noticed that the virtualtexturesystem is locking a lot of tiles at this time.
I’m not clear why the virutaltexturesystem is needing to lock the tiles, why that locking is taking a long time in other thread, and why the game thread is needing to block on those tasks finishing. I would have expected that if we opened the elevator doors too soon, we’d just see the level popping in, which we do. But I wouldn’t expect the game thread to stall.
Is it incorrect to set the LSV to loaded and visibilty? Should I set them to just loaded and then when everything is loaded, allow the elevator doors to open and set everything to visible? That just doesn’t seem correct.
Hi,
The behavior you’re experiencing is not entirely incorrect in the sense that the engine is doing what it’s designed to do—but it does reveal how virtual texture loading interacts with level streaming when visibility is enabled immediately.
When you mark your Level Streaming Volume as “loaded and visible,” the engine immediately makes the sub-level active and begins processing all its resources, which includes virtual textures. In your case, as soon as the player enters the LSV (i.e., steps into the upper floor area), the virtual texture system kicks in. This causes a burst of tile requests and locks the corresponding tiles so that they can be used by the renderer. FVirtualTextureSystem::GatherAndSubmitRequests can block execution of the game thread until these tile requests are processed - even if your texture pool isn’t oversubscribed. In short, the synchronous aspect of this operation forces the game thread to wait, causing stalls.
Because the doors are opening on a timer, the virtual texture updates are revealed “live” and if the system isn’t quite caught up with its background processing (tile locking and submission), the game thread stalls rather than simply popping the level in. This is why delaying the visibility activation (even if only by a frame or two until the virtual texture system has settled) can reduce stall times. Essentially, while the level might already be loaded (its actors are present, etc.), making it visible forces a flush of resources (like virtual textures) that can be a heavy operation if initiated at the wrong time.
So, it’s not incorrect to set the LSV to loaded and visible, but in situations like this (setting a level to loaded and visible immediately when the player enters a streaming volume) it can cause blocking of the game thread due to virtual texture locking. This blocking happens not because level streaming itself is synchronous but because making the assets visible forces the engine to synchronize some background tasks (e.g., finalizing virtual texture tile locks). These synchronous operations can cause noticeable hitches if they occur at inopportune moments.
A better approach may be to first set the LSV to loaded only (which will stream in the level’s assets without forcing the immediate processing of visual elements) and then, once all streaming - especially virtual textures - has settled, toggle the visibility (for instance, when the elevator doors open). This staged loading can decouple asset loading from their display, doing so can avoid sudden performance hitches. It’s akin to preloading resources and then fading or “popping” them in only when background processing (like virtual texture requests) is done. This way, you avoid having the game thread block on texture tile locks during gameplay.
Please let me know if this was helpful or if you have any further questions.
Thanks,
Sam
Hi,
it’s my understanding that level instances can use level streaming without using World Partition, but you will need to manually manage the streaming of the runtime-created level instances, as they will not be automatically handled by level streaming systems such as World Partition. So even if the sublevel can be streamed, the contained level instances will not automatically be streamed in the background when not using World Partition and may cause a hitch when the level becomes visible, which can block the game thread for many milliseconds depending on the number of level instances and actors that become visible at once (as mentioned here).
This article provides some in-depth information on level streaming, including optimization strategies (such as using soft object references to delay the loading of objects that are not immediately required) and some relevant CVars.
There are no specific CVars to log resources or textures being loaded on the game thread, but you might try
LogGameThreadMallocChurn.Enable (If > 0, then collect sample game thread malloc, re-alloc and free, periodically print a report of the worst offenders.)
r.VT.Verbose (Be pedantic about certain things that shouldn’t occur unless something is wrong. This may cause a lot of log spam 100’s of lines per frame.)
If you’re comfortable with C++, you can hook into asset loading functions like LoadPackage() or FStreamableManager::RequestAsyncLoad() and log when they’re called from the game thread. Combine this with IsInGameThread() to filter relevant calls.
Please let me know if the above helps.
Sam
Hi,
thanks for getting back and I’m glad to hear the links were helpful. Having the Level Streaming managed through World Partition will hopefully reduce the hitching. Please let us know how it goes.
Sam
Ok, this is what I suspect that the “visible” part was forcing the virtualtexturesystem to put all the tiles on the locked list and force their immediate loading.
I was able to play around with the LSV’s and make it so that a sub-level never goes from unloaded to “loaded and visible”. The sub-levels would go into LSV1 which was “loaded not visible”, then later to LSV2 with “loaded and visible”. And I could patiently wait in LSV1 to make sure all the sub-levels had finished their “loaded not visible” streaming then proceed to the LSV2 so that they would then go to “loaded and visible”. And I still got hitching.
In this case, I saw a large set of LevelInstances go from completely unknown to “loaded and visible” when I entered LSV2. So it looks like LevelInstances do not go to “loaded not visible” when their containing sub-levels are streamed into “loaded not visible”. And only when their containing sub-levels go to “loaded and visible” will the LevelInstanceActors get added to the world, which will immediately set them to “loaded and visible”. This article seems to say that LevelInstances are only supported in WorldPartition and so not supported with LSV: https://dev.epicgames.com/documentation/en-us/unreal-engine/level-instancing-in-unreal-engine. But maybe that’s too much assuming.
- Is there a way to have LevelInstances get “loaded not visible” when their containing sub-level does?
I removed all the LIs that were getting snap loaded visible to see if the hitch was just from them. It wasn’t. I still have a 378 ms hitch from just the normal streaming sub-levels. This is transitioning the sub-levels from “loaded not visible” to “loaded visible” and I get this hitch. Here is the callees graph from Timing Insights
UWorld::UpdateLevelStreaming 1 0.3780461 0.0000005 UpdateLevelStreaming Time 1 0.3780456 0.000008 LevelStreamingDynamic 8 0.3780376 0.0000184 STAT_UpdateLevelStreamingInner_OnLevelAddedToWorld 4 0.3733704 0.0038394 FGlobalComponentRecreateRenderStateContext::~FGlobalComponentRecreateRenderStateContext 1 0.3438686 0.0915858 AddPrimitive (GT) 30067 0.186406 0.0024559 UStaticMesh::GetUsedMaterials 27296 0.0342315 0.0335289 FMaterial::CollectPSOs 49013 0.0197491 0.0156446 Component CalcBounds 34931 0.0066635 0.00651292. is this expected to get this large hitch from instantiating the objects?
3. Is my only recourse with this being to chop up the sublevels more so they have less to instantiate?
4. can you recommend any cvars that would log out any resources/virtualtextures that are being rush loaded on the gamethread? If I had that list, I probably could more easily infer back to what streaming level they were coming from.
Those were helpful articles to look at. The first one wasn’t showing up in my searching for this topic, so thanks for calling attention to it.
I tried tweaking various streaming cvars to try to smooth out the streaming, but nothing could prevent the gamethread hitch.
Given that Level Streaming requires manually managing LevelInstance streaming and there is no non-destructive path to manually merging actors (these 2 things are the biggest issue with level streaming), we are now looking into World Partition, which has solutions for these issues. I’ll use City Sample as an example.