IRenderModule::LoadPendingVirtualTextureTiles - request to allow async loads (/submits)

Development Rendering
, ,
1

So to be fair we have experienced this back in 5.4- however looking at the code in 5.7 I don’t see this changed (though maybe it’s no longer relevant).

We implemented a widget that’s able to stream a virtual texture- this is useful for stuff like rendering large prerendered map images in the UI, etc. (and I would highly encourage Epic to ship something like this themselves- there is already code doing something similar in the World Partition Minimap widget (though last I checked the areas of the VT it was requesting to load were actually calculated incorrectly)- putting this into a dedicated UMG/Slate widget would be useful for everyone). As part of this we need to do the following to request the VT tiles for the texture being rendered to actually load:

ENQUEUE_RENDER_COMMAND(SVTImage_RequestVTTiles)(
	[InFeatureLevel, VTResource, ScreenSpaceSize, ViewportPositon, ViewportSize, UV0, UV1, MipLevel](FRHICommandListImmediate& RHICmdList)
	{
		// AcquireAllocatedVT() must happen on render thread
		IAllocatedVirtualTexture* AllocatedVT = VTResource->AcquireAllocatedVT();

		IRendererModule& RenderModule = GetRendererModule();
		RenderModule.RequestVirtualTextureTiles(AllocatedVT, ScreenSpaceSize, -ViewportPositon, ViewportSize, UV0, UV1, MipLevel);
		RenderModule.LoadPendingVirtualTextureTiles(RHICmdList, InFeatureLevel, true);
	})

This is pretty similar to code throughout the engine that tries to load VT tiles- however we have an extra param we have modded here-

virtual void LoadPendingVirtualTextureTiles(FRHICommandListImmediate& RHICmdList, ERHIFeatureLevel::Type FeatureLevel, bool bAsync = false) = 0;

If you follow the code down for LoadPendingVirtualTextureTiles you eventually get into FVirtualTextureSystem::LoadPendingTiles, which has a call to FVirtualTextureSystem::SubmitRequests that passes a hardcoded ‘false’ for bAsync. This causes LoadPendingTiles to generally hitch when requesting tiles- making this pretty unusable for runtime code. It’s trivial to just add a bool (and default it to false for back compat) to LoadPendingVirtualTextureTiles on whether or not the submit should be async (pass it down to FVirtualTextureSystem::LoadPendingTiles, which then passes it down to FVirtualTextureSystem::SubmitRequests).

Is this something Epic would consider adding themselves so we can remove our modification? It’s a simple modification (adding and passing a bool down a couple functions) that adds a bit of flexibility to the render / VT API and makes trying to manually load VTs at runtime a bit easier.

(Also, again, Epic should implement their own image widget capable of streaming a VT texture- but that’s beyond what I’m asking here.)

[Attachment Removed]

2

Steps to Reproduce
-Call IRenderModule::RequestVirtualTextureTiles to try to manually load or preload some VTs

-Call IRenderModule::LoadPendingVirtualTextureTiles to ensure the tiles get loaded

-Notice the call to LoadPendingVirtualTextureTiles likely hitches

[Attachment Removed]

3

Hi Brandon,

One change that happened between 5.4 and 5.7 is that we now support GPU virtual texture feedback in the UI pass. This may be interesting to you if the reason for using RequestVirtualTextureTile() was because it was your only option for making UI virtual textures resident.

But assuming that you still want to use RequestVirtualTextureTile()/LoadPendingVirtualTextureTiles(), what behavior would you want for the case of setting bAsync=false? It would avoid a hitch but may leave some pages not resident. Would your approach be to call the functions each frame while the textures are on screen, or for some fixed warm up time? That could be fine, but I wanted to validate the expectation here.

Best regards,

Jeremy

[Attachment Removed]

4

Hi Brandon,

Thanks for the extra context here. I can look at adding the async flag for the next release.

But also, given the mention of nanite, I notice that in the nanite API there is:

PrefetchNaniteResource(const Nanite::FResources* Resource, uint32 NumFramesUntilRender)

With NumFramesUntilRender this allows the streaming to be happen in the scene render update over multiple frames. A similar approach for VirtualTexture would allow us to call RequestVirtualTextureTiles() and allow the scene renderer handle the requests without any use of LoadPendingVirtualTextureTiles(). Would that be something that you would use?

Also how are you handling the nanite case currently?

Best regards,

[mention removed]​

[Attachment Removed]

5

Thanks for the background information. It’s really useful to have this extra context around the problems that you are needing to solve.

I agree that an actor-centric API can make sense for many situations. And there are a couple of similar thoughts that we’ve been hearing from other licensees.

I’ll add all this information to our backlog here for when we next revisit this.

Best regards,

[mention removed]​

[Attachment Removed]

6

For what it’s worth we’ve used it in both contexts- for the UI specifically we would call it each frame the widget rendered (which is needed anyways as the widget can pan the underlying VT image), we have also used it for warm up type code before where we are just trying to get materials resident before they are needed (such as before a cutscene (not one that is pre-setup such that we could use the existing VT request recording stuff)) and we couldn’t afford a hitch. All I was expecting in the case that bAsync is false is it continues to operate like it does now (though I assume that was an accident and you want to know what it does when bAsync is true since that’s when it might not be immediately resident). Essentially I’m, looking for this mod to be made:

[Image Removed](plus all the functions that pipe down to it passing the bool down)- the submit here already accepts a bool for bAsync, it was just hardcoded to false previously. I do understand however that it being async means they might not be resident immediately after the call so the function being named ‘LoadPendingTiles’ could be misleading.

UI supporting VT feedback does solve our need for the special VT widget which kinda eliminates our main need for this mod- but it’s still useful for us for warmups. I guess I would say even without this mod some official way to try to warmup textures on materials/actors (useful prior to a camera cut or cutscene) that doesn’t hitch would be appreciated / is desirable. In our use of passing bAsync as true even for warm-ups we have found it’s generally good/acceptable for getting VTs preloaded, even if not perfect. I guess for expectations I don’t really have any specific/I’m not asking the engine to do anything more than just allowing us to pass that bool down to the submit to not hitch- but I get wanting to know what we want at a high level and if you want the high level answer I would say (with UI solved) some way to preload textures for a material/actor would be nice- outside of the UI use-case this is what our warmup use care for this looked like.

void UXGameplayStatics::RequestVirtualTexturePreload(const UObject* WorldContextObject, const UMaterialInterface* Material, const FVector2D& InScreenSpaceSize)
{
	check(IsInGameThread());
	check(GEngine);
 
	const UWorld* World = GEngine->GetWorldFromContextObjectChecked(WorldContextObject);
 
	if (!World || !Material)
	{
		return;
	}
 
	ENQUEUE_RENDER_COMMAND(Gameplay_RequestVTTiles)(
		[InScreenSpaceSize, Material, FeatureLevel=World->GetFeatureLevel()](FRHICommandListImmediate& RHICmdList)
		{		
			const FMaterialRenderProxy* MaterialRenderProxy = Material->GetRenderProxy();
			MaterialRenderProxy->UpdateUniformExpressionCacheIfNeeded(FeatureLevel);
 
			GetRendererModule().RequestVirtualTextureTiles(MaterialRenderProxy, InScreenSpaceSize, FeatureLevel);
			GetRendererModule().LoadPendingVirtualTextureTiles(RHICmdList, FeatureLevel, true);
		});
}

While we’re here I would say we have a similar function to try to preload Nanite resources so similar functionality to try to preload a Nanite mesh would be useful.

Sorry if this wasn’t the answer you were looking for/I danced around what you want. My expectation is simply that bool gets passed down to the FVirtualTextureSystem::SubmitRequests function as it already supports it. But, again, at a higher level (and with UI solved in 5.7+) we would like ways to preload VTs for given actors/materials that are offscreen (and the same goes for Nanite if I can stick that request in there) as that is our remaining use case for this change.

[Attachment Removed]

7

We did use PrefetchNaniteResource internally to try to preload Nanite meshes and I think it generally worked- we did run into scenarios using it though where we wanted the resources immediately and the delay when calling it (even with 0 frames until render was passed) was problematic. We are probably fine with the current API there as long as it works. (Our specific use case where we needed the resources loaded _NOW_ was our title logo on the main menu in The Outer Worlds 2 was actually a 3D mesh and there was obvious pop in when the menu first rendered- we were fine hitching before displaying the logo (we were coming from a black screen anyways) in order to get it preloaded but nothing we did with PrefetchNaniteResource seemed to work (I think I even tried waiting a couple frames after the request but before rendering the mesh and it didn’t seem to work but it’s been a bit since I messed with it, part of the problem may have been the wonky state of the engine since we were starting it up and our main menu was the first map load)- I ended up giving up and we just un-Nanited the meshes for the logo to prevent the pop).

As requested, here’s the specific function that we had in our code to try to preload Nanite meshes:

void UIndianaGameplayStatics::PrefetchNaniteResource(const UObject* WorldContextObject, const UStaticMesh* StaticMesh, const int32 FramesUntilUse)
{
	check(IsInGameThread());
	check(GEngine);
 
	if (!StaticMesh || !StaticMesh->HasValidNaniteData())
	{
		return;
	}
 
	const FStaticMeshRenderData* SMRenderData = StaticMesh->GetRenderData();
	if (!SMRenderData || !SMRenderData->NaniteResourcesPtr.Get())
	{
		return;
	}
 
	ENQUEUE_RENDER_COMMAND(Gameplay_PrefetchNanite)(
		[NaniteData=SMRenderData->NaniteResourcesPtr.Get(), FramesUntilUse](FRHICommandListImmediate& RHICmdList)
		{
			GetRendererModule().PrefetchNaniteResource(NaniteData, FramesUntilUse);	
		});
}

A similar approach to VT loading as to how the Nanite prefetch is supposed to work could be fine too (just calling RequestVirtualTextureTiles and letting the renderer deal with it without using LoadPendingVirtualTextureTiles)- really both approaches have their use cases. Generally the reasons we have needed to preload meshes/textures in our games is we are starting cinematics, doing camera cuts in conversations (and we want all participants in the conversations to have their meshes preloaded- backgrounds aren’t as important), we rendered our map in our UI using a VT (as that just makes sense), or in the case of our main menu we are transitioning into a level and have a couple important meshes that we need to preload, even if it blocks (the main menu case). Obviously during gameplay we want most of these requests to be async/not block- but if we have fades into cinematics, etc. to cover block loads blocking in those cases is preferable to ensure everything is for sure ready before it’s needed.

I will say though, to take a step back for a second- these are useful lower level APIs but just outlining how/why we used them might be a good idea as I think our (and other licensees) lives could be made even easier/better by some higher level APIs on actors/components (especially with the UI VT thing solved). From a gameplay perspective it’s pretty rare (though still useful in some cases) to need VTs/Nanite for a _specific individual material or mesh_- which is how the preload functions work. Generally there is some important / relevant actor we want to preload (and preferably keep) it’s resources/etc. I’ve posted our preload functions we used on The Outer Worlds 2 but not well explained/shown (beyond the UI map image case) where we use this stuff specifically:

  1. We have an ‘actor stage’ class that is used to render an actor for a UI widget (via a scene capture)- this is used mainly in our inventory screen to display 3D renders of the player character and/or individual items. When we spawn the item on the stage we immediately try to preload the mesh and VTs to prevent pop rendering it.
  2. When entering a conversation (where we specifically frame characters closer up than during typical gameplay) we iterate all participants in the conversation, harvest all the materials off them, and try to preload all textures for all the materials on their meshes. (We probably don’t preload Nanite meshes on them because our characters are almost entirely non Nanite skinned meshes- once Nanite skinning is officially out this will almost for sure change).
  3. As already mentioned we were using RequestVirtualTextureTiles and LoadPendingVirtualTextureTiles with a specially crafted UI widget to do VT streaming for a VT texture it was rendering (and this shouldn’t be needed in newer engine versions).

[Image Removed]I would say, from our use cases- a more useful thing for us as an API would be the ability to mark components as ‘important’ in order to try to prefetch and keep texture and mesh resources for them in memory. I realize this is now a much bigger ask than the original request on this post- lol- but in terms of the challenges we had on our project, what we were using these APIs to solve, and what I think would be useful long terms/for all licensees- I think this would be the long term thing to try to implement.

I will say too, while I did focus on bumping priority on actors here, I do think there are use cases still for preloading resources for meshes/materials/etc- a basic use case I can think of where this would be useful on The Outer Worlds 2 would be trying to preload the meshes for weapons before equipping them on the player (however having said this our weapons did heavily use VTs and Nanite on TOW2 and this never really came up as a problem- but it’s a good demonstration of where I think an API operating on individual meshes/materials could be useful/solve problems in titles).

Sorry for the long response- hopefully you have a good idea of what our challenges were/what we were trying to solve. Thanks for all the help so far!

[Attachment Removed]