Implementing Image-Based Variable Rate Shading

[privateeyelab]

I’m trying to use an arbitrary image with screen-space variable rate shading. I’m unfamiliar with Unreal’s rendering pipeline, but it seems that it has to do with render graphs.

I’ve done this with Godot since it has an explicit variable to set the texture to perform VRS. This page documents how it functions in Godot, and this API reference shows how easy it is to use by setting two variables.

Unreal seems to be much more involved since I can’t find a single editor reference to it beyond materials (which is insufficient for my needs since it is not the entire viewport). I understand that this feature is very new and just released in UE 5.4 which is what I’m using. How can I arbitrarily use and change the image for screen space variable rate shading? Any help would be appreciated.

[ClockworkOcean]

I haven’t watched this yet, any good?

Variable Rate Shading in Unreal 5

[privateeyelab]

Thanks for the fast reply. Unfortunately, this person showed only how to do uniform VRS on a texture. I’m looking for how to implement Tier 2, screen-space VRS, like how foveated rendering works. I’m looking for some way to pass an image that has (non-uniform) shading rate information on it.

From this page on the Unreal website, the fixed foveated rendering can use an image like this to tell the hardware VRS how to function.

So far what I’ve found is that these images are generated on the fly in the source code, still working on it.

[ClockworkOcean]

Yes, I looked into it a bit more. It seems like it’s either material based, or used for VR.

[privateeyelab]

Here’s an article for anyone interested in an example of how such tech can work. It seems that game studios generate a image based on scene data, but what I’m interested is the particular mechanism in Unreal to set this image.

[privateeyelab]

I was able to implement VRS from a given image! I’m currently working on a more robust plugin to make it easier for other people to use this in their projects, but here is a high-level overview of how Variable Rate Shading is implemented, and where to begin for anyone who is interested/wants to roll their own. You will need to dig around in the source files quite a lot in order to get this working, so refer to the filenames below to find examples on how to do this.

Also, if you’re not interested in implementing your own VRS feature, scroll down to the “Console” section to see how to activate Contrast Adaptive Shading and Foveated Rendering that Unreal comes with.

Structure
Unreal 5.4 (not sure about earlier versions) has two important classes defined in the public VariableRateShadingImageManager.h include file under “Source/Runtime/Renderer/Public/” in the file system. The private files that show implementations of these classes can be found in the “…/Renderer/Private/VariableRateShading/” folder.

The two classes of importance are:

• FVariableRateShadingImageManager: The class that manages all of the VRS internals. (We’ll call this the ImageManager.)
• IVariableRateShadingImageGenerator: An abstract interface that provides the images to the above class. (We’ll call this the ImageGenerator.)

The ImageManager contains arrays of registered and active ImageGenerators which must be registered through its function: GVRSImageManager.RegisterExternalImageGenerator(). Note that while you can register as many ImageGenerators as you want, only two will be active at any one time. This is a result of the ImageManager’s code, of which modifying is outside the scope of this post.

The purpose of the ImageManager is to take images that the ImageGenerators pass to it, and use those for VRS. Thankfully, that means the really difficult part is already taken care of, so all that needs to be done is implement a custom ImageGenerator to do whatever we need.

ImageGenerator has quite a few functions to implement, but most of them are self-explanatory in the code, with some helpful comments. To see how to implement these, the FoveatedImageGenerator.cpp file is a great help to see how to implement these functions and use a Compute Shader to generate an image like the ones earlier in the thread without texture data. In my implementation, the custom ImageGenerator is a global object so other functions (e.g. Blueprints) can use it.

Implementation
I tried to write the code similar to how Godot implemented their VRS texture system. That is, take a texture that has specified color value and write values into another texture that the ImageManager can use properly. The particular VRS values can be found in the RHIDefinitions.h file in the EVRSShadingRate enumeration.

Particularly in the PrepareImages() function of the custom ImageGenerator, a somewhat “high-level” example to accomplish the goal of a custom texture would be:

1. Load the particular input texture as an FRDGTexture2D so that the Render Dependency Graph can use it in the Compute Shader.
2. Create another FRDGTexture2D and corresponding FRDGTextureUAV as a render target to for the Compute Shader and store the FRDGTexture for the ImageManager to access via GetImage().
3. Setup and dispatch the Compute Shader to generate the image and then store the image for GetImage() to return.

Don’t forget that you have write the Compute Shader and its corresponding C++ class. The example VRSShadingRateFoveated.usf (located in “Shaders/Private/VariableRateShading/”) is particularly helpful and simple compared to the others. Reading through the source code is essential to getting a custom implementation completed and functional.

After this, the only difficult part left is to figure out how to set that texture. I used an external Blueprint library to set the image and added an extra function to the ImageGenerator to set the texture to use. Make sure that you implement IsEnabled() somehow so that the ImageGenerator knows that this algorithm is active (AND AFTER YOU SET A TEXTURE OR IT WILL CRASH).

Console
To activate any of the VRS features, the easiest way to do it is through the command console in the Unreal Editor. These settings persist even in play mode, so use them to get things configured. A very useful command for debugging is r.VRS.Preview 1 which will turn on the colors to show which areas of the screen are shaded at what rates. You can try this with r.VRS.ContrastAdaptiveShading 1 to try out an algorithm that shades at a coarser rate where contrast is low, so it is very difficult to see without the preview. The source file ContrastAdaptiveImageGenerator.cpp is also very helpful for implementation help.

Hopefully this helps someone understand how to use VRS in Unreal, and implement their own algorithm if needed. These features are not visible anywhere except the console, so even if you didn’t want to roll your own VRS features, you can still take advantage of Contrast Adaptive Shading or Foveated Rendering. If anyone has any questions, I’d be happy to answer them.