Debugging USF (Unreal Shader Files)

Development Rendering
1

Hello!

I’m looking for information on how people go about debugging USF shaders - including when they are not working, but also if they are not compiling (where can one get information on why they are failing?).

Edit: I’ve found this document, which is a good start!

2

For debugging during compilation, If you are running your engine from VS and it fails to compile a shader, it will tell you in the output window of VS and even tell you what line is the culprit. (Wasn’t aware of the r.ShaderDevelopmentMode though, that will make things much quicker)

For debugging during execution. I have tried to use the VS tools for graphics debugging with very little success. If I can even get it to capture a frame, it will crash when trying to reload that same frame. So if you or anybody else finds out any more info about debugging during execution, I would be very interested.

If your working in Windows with D3D then dont forget to add -d3ddebug to the command line to get more info from the D3D pipeline. I spent hours trying to figure out why I couldn’t create a Stream Out Geometry shader until I found that command line.

Thats about all I can add, I’m sure most of it is just repeated info that can be found in the above link.

3

Thanks - all handy information. Hopefully someone’ll have a decent idea of how to debug it with something like the VS tools.

4

I have never had any success with the various shader debugging tools, they seem to be tested in small apps only.

My debugging workflow consists of outputting intermediate values to a render target and then looking at that texture with the ‘visualizetexture’ or ‘vis’ command. It can get difficult when the shader gets complicated.

5

Thanks - that’s more or less what I’ve been up to, but having a decent tool to work with would be infinitely more helpful :confused:

6

Just for future reference, I just found out about Crytek’s RenderDoc, so thought I would try that out, and with a small change to the code (In RenderDoc) I have been able to use it to do frame debugging in UE4.

7

For my part, I was able to capture frame with RenderDoc and with Nvidia nSight. It work very well, both. (With a little preferences for Nvidia nSight, because it’s pluged in VS).

But I can’t have HLSL code out of my shaders. (I modified the flags of ShaderFormatD3D to have DEBUG_SHADERS 1, it should therefore compile shaders with debug info, but I got the shaders in ASM…)
If someone have a solution to have the debug symbols, it would be great !

8

Sorry, I don’t have a solution.
I’ve tried to capture debug data with pix and with renderdoc, but without success. Actually, with renderdoc I can run the editor lobby (where I need to select a project) and nothing more - on project select editor closes without any errors. Renderdoc logs have no error info too. So how do you using it?

Upd.: I packaged a game and now can run renderdoc with it. But I don’t know - packaging happens in release mode, so maybe where’s some data loss? Another sad side - in order to debug shader I need to build a project what takes 15 min on my machine. Oh, and no hlsl too, just asm, but it’s better than nothing.

9

Hello,

Okay, maybe I’ll make a tutorial for this; I found a way to have HLSL code in nvidia nsight, with renderdoc, it doesn’t work (even if it should…).
Actually, UE4 strip debug information from the compiled shaders in the UE4 shader compression process. I found how to add debug information in the shader, but I didn’t see that the after the compilation, the shaders was compressed, and farewell debug informations.

I have another problem now, but it’s very probably because my hardware doesn’t allow it, more than a problem with nsight or UE4 -> I can’t make breakpoint in shaders. But well, I can have the output of UE4 (there’s actually two way to have them), and I can have some information in the pipeline with nsight. It’s enough for now, but I’ll probably check on the nvidia website if the breakpoint problem come from my hardware (I don’t even meet the UE4 minimum shame), or if the compilation process omit some informations.

EDIT :
I can confirm the problem come from my configuration, as stated on the nvidia website.

So I’ll try later, if I need, to remote debug my UE4 project on another computer, from this one.

Anyway, if someone need it, you should now be able, with a remote configuration, to set breakpoint in a shaders. :slight_smile:

PS : I’m not english speaker/writer. So, if you found some big mistakes, please, let me know.

10

I haven’t tried source in Nsight for a while, but some things you can try:

  • Remove the call to D3DStripShader in D3D11ShaderCompiler.cpp. You have to recompile ShaderCompileWorker Win64 Development to propagate the change. And then add a space or something to common.usf to force shaders to recompile (if you are working on a global shader just edit it a bit).
  • In UE4 we compile the shaders in a different app (ShaderCompileWorker) which allows making use of all your cores. However tools like NSight don’t know about this other app and they only hook the compile calls in the main executable. If you open BaseEngine.ini and set bAllowCompilingThroughWorkers=False, any future shaders will be compiled from the main executable. Of course, this will make compiling super slow.
11

Thanks,

That’s exactly what I’m doing, but I didn’t add a space in common.usf. This could explain why I only got my Vertex Factory source in nsight and not the others ?
And I can try for removing the compiling with worker. (Already did too, actually). But, as you said, it’s very slow. And (I guess) as I’m working on a VF, every times I make a change to my VF it need to recompile all the shaders (around 4000 shaders…).

With the worker, it takes something like 10 minutes on my computer. So without, it will be long…

In annex, sometimes my VF compile in less than a seconds (it just recompile the VF ?), sometimes it recompile the whole… I don’t really know why.

12

>With the worker, it takes something like 10 minutes on my computer. So without, it will be long…

Yeah that would be murder

If you are just making usf changes to your VF and not C++ changes, you can setup a simple test level, run it in the editor, then recompile just a single material applied to a single mesh (or whatever your vertex factory is). When you recompile a material in the material editor, it re-reads all the usf files from disk so you can quickly iterate that way.

When you are making a new vertex factory be sure not to just return true from the ShouldCache. That will make it compile the shaders everywhere on every material. Add a usage flag to the material, then artists have to opt-in to using it. Actually the editor will automatically set the flag when possible. Then only those flagged materials will compile your VF, which can cut down the shader combinations needed massively. Have a look at any other vertex factory for a working example (except local VF).

Probably because shaders are cached using a hash of the file contents. If you make a change, recompile, then undo the change, recompile, the second recompile will be very fast (seconds) because the results are already cached in the Derived Data Cache.

13

Yes, that’s make sense, thanks for your input .

Actually, I WAS looking at Local VF. Until I understand what it make. But well, I’ll change my ShouldCache methods.

Thanks again !

14

Hello!

I do not mean to necro this thread, but I thought my small tutorial on this topic might be useful for people who find this thread but are still wondering what tools can be used and how.
Here is the link:

I hope this will be useful to someone :slight_smile:

Best regards,
Temaran

15

I’m new to developing shaders using USF, but I would say, try creating a shader first using shadertoy.com and translate the GLSL code over to HLSL once you are satisfied with the results that you created on shadertoy.