Hello again everyone!
Another weekend has arrived and I thought I’d spend some more time trying to figure this out. However I must admit I feel I am running out of ways to debug this. I will spend the day trying to fiddle around with already working render targets to try and gain more insights into how they work so I might be able to then make my new one come alive.
I thought I would summarize my current problem though in the hopes that someone might be able to provide insight, and in the worst case as documentation 
My current progress is publicly visible on my fork listed in a previous post if anyone would like more context.
As you might know, my custom render target is set up by copying and modifying the code for the custom depth buffer. So my render process starts in DeferredShadingRenderer.cpp, to SceneRendering.cpp, on into my new CustomTagRendering.cpp and finally resolves in my Drawingpolicy in TagRendering.cpp.
I’ve currently set up my render target in SceneRenderTargets.cpp (it shows up in the Vis/VisRT console commands) and I’ve also exposed it through the buffer visualization combo box in the editor.
No matter what I do though, my output is always a black texture. I’ve tried flipping through most texture formats, tried turning depth stencils off with
RHICmdList.SetDepthStencilState(TStaticDepthStencilState<false, CF_Always>::GetRHI());
(I’ve tried cf_never too, not sure if the stencil is expects a positive or a negative to skip the fragment)
Clearing the buffer with:
RHICmdList.Clear(true, FLinearColor(1.0F, 0, 0, 1.0F), true, 0.0f, false, 0, FIntRect());
I have tried this on other buffers too (like basecolor), and it doesn’t work there either, so I can only assume that it is done at a later stage by the engine overwriting my call. Anyways, I thought I’d mention it for completeness.
My drawing policy declares a simple pixelshader, vertexshader and no-op hull and domain shaders:
IMPLEMENT_MATERIAL_SHADER_TYPE(, TTagOnlyVS, TEXT(“TagBufferShader”), TEXT(“MainVertexShader”), SF_Vertex);
IMPLEMENT_MATERIAL_SHADER_TYPE(, FTagOnlyPS, TEXT(“TagBufferShader”), TEXT(“MainPixelShader”), SF_Pixel);
My usf files are basically only pass through and forcing the output to be a certain color:
Vertex:
struct FLightMapDensityVSToPS
{
FVertexFactoryInterpolantsVSToPS FactoryInterpolants;
float4 WorldPosition : TEXCOORD6;
float4 Position : SV_POSITION;
};
void MainVertexShader(
FVertexFactoryInput Input,
OPTIONAL_VertexID
out FLightMapDensityVSOutput Output
)
{
FVertexFactoryIntermediates VFIntermediates = GetVertexFactoryIntermediates(Input);
float4 WorldPosition = VertexFactoryGetWorldPosition(Input, VFIntermediates);
float3x3 TangentToLocal = VertexFactoryGetTangentToLocal(Input, VFIntermediates);
FMaterialVertexParameters VertexParameters = GetMaterialVertexParameters(Input, VFIntermediates, WorldPosition.xyz, TangentToLocal);
WorldPosition.xyz += GetMaterialWorldPositionOffset(VertexParameters);
Output.WorldPosition = WorldPosition;
ISOLATE
{
float4 RasterizedWorldPosition = VertexFactoryGetRasterizedWorldPosition(Input, VFIntermediates, Output.WorldPosition);
Output.Position = mul(RasterizedWorldPosition, View.TranslatedWorldToClip);
}
Output.FactoryInterpolants = VertexFactoryGetInterpolantsVSToPS(Input, VFIntermediates, VertexParameters);
OutputVertexID( Output );
}
Pixel:
void MainPixelShader(
FVertexFactoryInterpolantsVSToPS FactoryInterpolants,
float4 WorldPosition : TEXCOORD6,
OPTIONAL_IsFrontFace,
out float4 OutColor : SV_Target0
)
{
OutColor = float4(1.0, 0, 0, 1.0);
}
Debugging the code, I can see my drawing policy being run on the correct actor/mesh with the WorldGridMaterial (since my usf-files are in the Engine/Shaders/ folder I assume this is the correct material as my current hypothesis is that all such shaders are compiled and loaded into this “master material”. Debugging for example the custom depth buffer functions reveals usage of the same material.
The draw operations all run without errors, and I can see that I get all the way down to the RHICmdList.DrawMesh hardware function calls.
My current hypothesi are therefore these ones:
- My shaders are not doing what I think they are doing
- My drawing policy contains other errors
Update: Disproven:
XXX The render target is actually drawn to, but the reference VisRT is referring to is actually not the correct one
XXX My draw operations are being cleared / drawn over after they are completed
XXX My render target is being recreated without my knowledge
The problem is that debugging all of these requires a graphical debugger. I have so far tried using RenderDoc (crytek), nSight, VS Graphical debugging tools and PIX, all of which either crash (all except renderdoc) or do not do anything (renderdoc) when I try to capture a log. I’m assuming this is because of how the editor is designed (Im guessing all these tools look for a primary rendertarget to analyze or similar, and the editor does not expose this in a conventional way to make it easier to iterate on your project).
The recommended way to debug this according to several posts on the forum is to output intermediate results into other rendertargets, but as you can see, I can not do this either since I still do not have access to drawing to any target so far.
I had another idea of just trying to draw a line using DrawPrimitive, but I abandoned that idea when I realized there is no simple way of binding shaders using RHICmdList without using the vertexfactory system. And as the point of the test was to make a minimal sample, it kind of defeated the purpose. I might revisit this idea though if I don’t get anywhere when playing with the other targets.
It is very unfortunate that UE4 does not support any graphical debugger as this would have made this a lot easier, even having a command that outputs all operations as a log would help me. I have tried using -d3ddebug from Graphics Programming for Unreal Engine | Unreal Engine 5.1 Documentation but this only provides extremely bare bone init messages and does nothing for getting a debug output on what is happening with my render target. It also makes the editor crash when running Vis x in the console.
I have also tried setting r.DumpShaderDebugInfo from Shader Debugging Workflows Unreal Engine | Unreal Engine 5.1 Documentation
but this only dumps shader compile and logs and still gives me no runtime information.
I think this summarizes my last 15~ hours in the engine code fairly well, but I might update the post if I remember more details
----- UPDATE ----
I managed to get a different system to render to my render target without problems, so this only leaves the hypothesis that my shaders (or some other part of my drawing policy) contains errors. That definately cuts down on the work, but it also means that the remaining hypothesis is basically a black box since I’m still unable to get graphical debugging to work. I might return to trying to draw some simple geometry from my policy to start with. The other option is to debug yet more working engine code to try to figure stuff out. It will probably end up to be a combination of both
Best regards,
really impressive documentation of your work, I’m sure many will find your this very helpful. Keep the work up!!!
