[Foveated Rendering] Pupulate GBufferA in Prepass & Offline Texture Preprocessing

Hi everyone,

I am currently working on my master thesis which is about Foveated Rendering in Unreal Engine.

I am at the very beginning and I currently try to implement the method from this Paper as a reference method.
The core idea is to save some performance by creating a probability distribution and sample from that. Then utilizing Early Z-Culling the Shaders will be invoked only on sampled pixels. The rest is then interpolated efficiently and presented to the user.

So far I have managed to create a simple distribution only based on Visual Acuity and a binarized Gradient of the Depth Texture (right before the BasePass). When a sample is generated I add a small Delta to the Depth-Texture which prevents the Basepass from rendering non-sampled pixels:
FoveatedRenderingSampling.png

Right before SceneColor is resolved the pixels are interpolated with an algorithm called ‘Pull-Push-Interpolation’ resulting in the final Scene (FeatureMap is a visualization of the propability distribution used for Sampling):

Now I have the problem that I need to access the WorldNormals for the Silhouette-Part of the distribution.

So is it possible to move WorldNormal-Rendering from BasePass to the PrePass?

I haven’t looked too deep into the rendering code so far and unfortunately my Computergraphics and C++ knowledge is rather basic than advanced. But I take up this challenge so far :wink:

Another thing is that I will need Saliency-Textures in the future which should ideally be precomputed offline. Can anyone give me an advise where to start looking for possibilities to realize that? Maybe by modifying the Editor to compute a Saliency-Texture when a Texture is loaded or saved?

Thanks in advance!

I hope everything is clear, otherwise I will try to explain it more clearly.

Best,
Ceron

Edit: I noticed that TextRenderActors seem to not ne present in SceneColor when SceneColor gets interpolated. Any Ideas how to fix this?

Okay I managed to realize Normal-Rendering in PrePass.
If you compare my results with the reference there are some differences:

[table=“width: 500”]

Reference
PrePass


![ReferenceNormal.png|1206x572](upload://cmpk4KhPkB5JqjwbULWIcxyCPUc.png)
![PrePassNormal.png|1195x572](upload://ygVE8VJBhGMGnliRh7u3Hwg89uc.png)

How did I do this?

  • Modify second branch in FDepthDrawingPolicyFactory::AddStaticMesh such that PositionOnly buffers will never be used (bad idea, because this is an optimization, but I was unable to change it to a ‘PositionAndNormalOnly’-Buffer so far - any suggestions?)
  • Move GBuffer-Allocation in FDeferredShadingSceneRenderer::Render right before RenderPrePass() is called
  • Override bNeedsPixelShader in FDepthDrawingPolicy’s constructor with true to force the factory to use pixel a pixel shader
  • make FDepthOnlyPS::ShouldCache() always return true to make the shader available in the ShaderMaps
  • Change BlendState in SetupPrePassView() to TStaticBlendState<CW_RGBA>::GetRHI()
  • in FSceneRenderTargets::BeginRenderingPrePass() Set ColorTarget to GBufferA->GetRenderTargetItem().TargetableTexture
  • in DepthOnlyPixelShader.usf and DepthOnlyVertexShader.usf I had to redefine MATERIALBLENDING_SOLID to 0

Finally I take the Normal from MaterialParameters.WorldNormal and do the same as in EncodeNormal() in DeferredShadingCommon.usf: OutColor = float4((Normal0.5+0.5),1.0)*.

I am not sure about the sideeffects of these modifications (particularly relating to performance which is crucial for my thesis, but with no alternatives I will use it this way and compare everything using this modifications).

Does anyone know why these differences are there?

Hopefully someone could help me out…

Best,
Ceron

Just wanted to say that this looks awesome. I’m assuming you’re aiming for improved VR performance with the project? Any idea of what kind of performance savings would be possible?

Thank you! Yes I am aiming for better performance and also not creating any perceivable differences. In the linked paper the authors state they achieved an overall reduction of rendertime by roughly 25% in their non-optimized prototype. These savings depend on the per pixel shading cost. In my own method I’d like to address geometry as well (e.g. by using lower LOD-Levels in the visual periphery but I am not sure yet) to further improve the performance.

Because Unreal Engine is highly optimized already, I think the possible savings will be (much) smaller and I’d have to guess what kind of savings would be possible. I am very curious about how much performance could be saved and try to do my best (obviously :smiley: ), so let’s see

Just here to say “Awesome!”.

Just in case anyone else could be helped out with these information: The problem is now kind of solved!

Comment out the following code section in void FDepthDrawingPolicyFactory::AddStaticMesh(FScene Scene, FStaticMesh* StaticMesh)*



if (!Material->MaterialModifiesMeshPosition_RenderThread())
{
	// Override with the default material for everything but opaque two sided materials
	MaterialRenderProxy = UMaterial::GetDefaultMaterial(MD_Surface)->GetRenderProxy(false);
}


You then need to make sure that the DepthOnly VertexShader will be available in the ShaderMap. So I added the following at the beginning of TDepthOnlyVS::ShouldCache(EShaderPlatform Platform,const FMaterial Material,const FVertexFactoryType* VertexFactoryType)*:



FString FactoryName = VertexFactoryType->GetFName().ToString();
if (FactoryName.Equals(FString(TEXT("FLocalVertexFactory")))) {
    return true;
}


In void FStaticMeshSceneProxy:: DrawStaticElements(FStaticPrimitiveDrawInterface PDI)* there are created many instances of FMeshBatch which will be inerted into the StaticMeshDrawLists together with the corresponding DrawingPolicies. For the DepthOnly Pass this is done by use of GetShadowMeshElement() which I now replaced with GetMeshElement().

Here is how it looks now:

hey,

any update in the foveated rendering for google vr?