Download

[Tutorial] Pixel and Compute Shaders in UE4

Hello everyone!

As always, I’m not sure if this post fits best in the rendering forum or the C++ one, but I chose C++ again because it might be slightly more relevant.

It’s been a while, but I’ve put together a new tutorial. This time on how to get custom HLSL pixel and compute shaders to work in your project.
This essentially lets you use shaders completely unrestricted by the Editor’s material designer, which is awesome, don’t get me wrong!
It is just that it will not let you run compute shaders or output your HLSL pixel shader to some arbitrary TextureReference :).
This is something that I’ve heard friends ask about, and I didn’t really think there was any good resource for how to get up and running.

You can check out the tutorial project on my github:

It is important to note that getting this to work will require the project plugins that I provide as well as copying two shader files to your Engine/Shaders folder.

I will post the .md in the next post if you want to read about the details but are too lazy to click the link :wink:

—update—
I have now updated the project with a third plugin that will automatically copy the required shader files to the engine/Shaders folder, so you don’t have to remember to do this every time you update the shader file.
I am working on an engine pull request that will let you simply have your shaders in your Project/Shaders folder, and the engine will recognize this, but in the meantime, this should do.

The plugin also removes the files again when you close the engine to try to be a good citizen.

/Temaran

This project is a tutorial project for how to create shaders in UE4. Most material effects can be created in-editor using the excellent tools that Epic has provided us with. There are some times though, where you simply want to use a pixel or compute shader to do some work but you don’t want this work to end up in a material surface or post process material, but simply have access to it in a render target, a Cpp texture obect or a struct. This is where basic HLSL shaders really shine. And while it is possible to use normal HLSL shaders in UE4, there doesn’t seem to be any tutorials for it yet, and there are some complications that make it necessary to take a few detours before you can actually create and run your shader. It is worth to note that this is not a tutorial on how to program shaders in general, or how to write HLSL, but rather how to get shaders working in UE4. For learning how to program shaders or HLSL, I recommend other resources, such as MSDN: High-level shader language (HLSL) - Win32 apps | Microsoft Docs or instructional books for more advanced users such as “GPU Pro” by Wolfgang Engel.

The HLSL: There is no simply way to compile a shader and have it work properly like you can in a simple project. I haven’t investigated it much, but compiling your shader from an inline string or loading a file and compiling your shader code does not seem like it is an easy task in UE4. Instead the recommended way is to place your shader code in so called *.usf files under your engine’s Engine/Shaders/ directory. Depending on if you are using a binary or source build of the engine, this will obviously change which Engine/Shaders/ folder you actually have to use.

The shader compilation: When you have placed your shader code in the correct folder, you need to compile it. This is done using a macro in your Cpp source called IMPLEMENT_SHADER_TYPE. One problem with this though is that if you simply place this and your other code in your main project, you are going to have problems since this macro must be available and run during an early phase of engine start up, and in addition to this cannot be rerun if you decide to recompile your code in-editor. Ignoring this will actually crash the engine :). The solution is to put the shader declaration and initialization into a plugin and override the load phase of the plugin in question to “PostConfigInit”.

Shader parameters: Shaders take their input from the generic ParameterMap system in UE4. You can program this from your shader declarations using a combination of macros and code calls. The macros are generally used to declare structures to be used in the shaders, such as uniform / constant buffers etc. And the code parts are to bind runtime instances of these buffers to the ParameterMap. You can also bind resources like textures to the shaders if you want to.

Shader invocation: When you have declared your shader in the appropriate plugin, you can now start up your project and start using it. Shaders in UE4 have to be invoked on the rendering thread, to hopefully nobody’s surprise. To make sure we get our code to run on this thread, Epic has given us another handy macro called ENQUEUE_UNIQUE_RENDER_COMMAND_ONEPARAMETER. There are variations of this macro that let you send more than one parameter too, but since we only want to send ourselves, one parameter does fine. We simply instruct the render thread to take a pointer to our shader consumer object and call a function on it. It is inside this function that we can do all the appropriate calls to set rendering states, buffers, shaders etc. as well as dispatch rendering calls. All rendering functions are accessible through an abstraction layer called the RHI “Rendering Hardware Interface”, which as the name suggests is a way for our code to be platform independent. Most function names on the RHI seem to be inspired by directx rather than any other API though, which might be good to know. If you are having trouble finding a particular function, the best way is therefore to check MSDN before any OpenGL resources or otherwise.

Shader output: After you have run your shader it is of course time to harvest your output. There is no special UE4 magic to this step as it uses the normal SetRenderTarget RHI functions for pixel shader output and your UAVs for compute shader output.

How to run this project:
To get the project to run, you first need to bind it to an engine, you do this like any other project by right clicking the uproject file and switching version if needed, and then doing Generate Project.
You then need to open up the solution file and build the source code.

How to use this project:
I would recommend checking out the example project supplied in this repository to understand how to best use the project code. All the relevant files in the repository are:

  • Everything under the Plugins/ folder (For declaring the shaders)
  • Everything under the CopyToEngineShaders/ folder (For copying into the Engine/Shaders/ folder, these are the actual HLSL shaders)
  • Source/ShaderPluginDemo/ShaderPluginDemoCharacter.cpp/.h (These are changes and additions to the main character class in the demo project to allow for the shader’s use. I have removed all non-related comments in these files to make it easy to see where I have made changes)
  • Everything under the Content/ShaderPluginDemo/ folder (These are the editor objects that I use to set up the shader use in the scene)
  • The project settings file (I have created some new input bindings)

Project controls: W/A/S/D - Movement Space - Jump Move mouse - Look around Left mouse button - Paint the object you are aiming at with the output of the compute shader/pixel shader chain Q/E - Change the blend amount on the pixel shader. Q moves the blend closer to the pixel shader simple gradient, while E moves the blend closer to the compute shader output.

General comments on the demo project: The demo project demonstrates how to use both pixel and compute shaders using the method I have explained. In the tick function, I first run a compute shader that generates a cool visualization of a galaxy entirely within the shader. This is then packed into a R32_UINT format to allow for usage within the pixel shader which is then run that simply generates a gradient and then blends with the compute shader output. The user can shoot objects in the demo scene to apply a dynamic material instance to the object hit that is then loaded with the rendertarget containing the pixel shader output. You can also modify the amount of blending that the pixel shader does by holding the Q and E buttons.

I hope someone finds this useful :slight_smile:

Best regards, Temaran

I also added an entry to the wiki:

This needs more love, thank you very much Temaran for your hard work, I think this will be very useful to the community. On to check your sweet plugin I guess!

Hi Temaran, tried launching your ShaderDemoProject, doing your step by step instructions but I get this error

Fatal error: [File:C:\Users\ernesernesto\Documents\Projects\UnrealEngine4.6\Engine\Source\Runtime\Core\Private\HAL\FileManagerGeneric.cpp] [Line: 613]
Invalid BufferCount=0 while reading …/…/…/…/…/…/Desktop/UE4ShaderPluginDemo-master/UE4ShaderPluginDemo-master/Content/ShaderPluginDemo/PlayerCharacter.uasset. Pos=3655789018, Size=72724, PrecacheSize=2147483647, PrecacheOffset=3655789018

I’m using unreal 4.6 in Development Win64 configuration

Seems there are something wrong with the PlayerCharacter.uasset :frowning: preventing me to launch the project

Oh man! Thank you so much!

Hello ernesernesto!

I completely forgot to mention that the plugin is built for 4.7!
I usually put this into my branch names to avoid confusion, but for some reason I must have forgot this time, will fix that right away :slight_smile:

I’m not sure if this is what causing the problems for you. I will try to run it on a second computer to see if I messed something up, it might be some file corruption or something else.

It is worth reiterating though that it is made for 4.7. There are some bugs in 4.6 related to setting material instance dynamics to meshes and not having them update correctly which was resolved in 4.7 which is why I chose that version and not 4.6 :slight_smile:

I hope this clears some problems up! :smiley:
I’ll get back after I’ve tried it on my work compy :slight_smile:

/Temaran

Hello!

I just pulled and tested on my work-compy, and what do you know! For some reason some Joystick header was included in the character source. I have no idea why it was there, or why it is working on my hobby compy, but that’s computers for you eh?
I also added a 4.7 branch to the gitrepo and synched master with it, so it should be more clear now that you need to use a 4.7 build for this to work :slight_smile:

I hope people can get it to run without too much trouble :slight_smile:

/Temaran

Hello again!

I have now updated the project with a third plugin that will automatically copy the required shader files to the engine/Shaders folder, so you don’t have to remember to do this every time you update the shader file.
I am working on an engine pull request that will let you simply have your shaders in your Project/Shaders folder, and the engine will recognize this, but in the meantime, this should do.

The plugin also removes the files again when you close the engine to try to be a good citizen.

I hope this simplifies things.

/Temaran

Hey this looks great Temaran! Thanks a lot for the tutorial/sample on this topic, it is something I have been meaning to look into further. :slight_smile:

Thank you for the kind words!

I try to write tutorials at least semi-regularly :slight_smile:

If anyone has an interesting topic that they would like me to look at, I welcome suggestions :slight_smile:

/Temaran

Hi Temaran, your work help me so much, thank you! Would you like to extend this plugin (with your approach) for Hull and Domain shaders and also the tessellator :slight_smile: (or perhaps in different post :p) I believe you will make the path to the bright side :slight_smile:

Hello P2ssw0rd!

That’s definitely something I might look into. I’ve been playing around with the tesselator pipeline step a bit in one of my UE4 pet projects, and was working on a tutorial for that, but I haven’t touched it for a while now, so I might just make a simpler version and include in this project.

Best regards,
Temaran

Any news on this? Would love to see shaders in the local projects directory…

Hello!

I’ve actually completed a pull request to add support for this. I just haven’t… err… posted it O_O
I wanted to do some more QA and go through the code again after I’d cleared my head from all the details before I made the official pull request and posted on the forum regarding it, but the source is publically available to people who have source access (which is everyone now I suppose?) on:
https://github.com/Temaran/UnrealEngine/tree/ProjectLevelShaders

As it is a branch though you will need an engine source project for it to work.
I have rebased all the work into one commit on that github branch to make it easy to integrate / cherry pick into other branches without much hassle.
Hopefully Epic likes it enough to accept the pull request when I get around to it though :slight_smile:

Also, this is just the first of a series of pull requests I aim to make to eventually make my tag buffer work available as a plugin instead of as an engine branch.

— edit —
I actually realized it’s pretty silly to sit on it, as I’ve been doing for a couple of weeks, so I created the pull request:
https://github.com/EpicGames/UnrealEngine/pull/1218

I’m still at work now, but I’ll add more data to the request when I get home. There is quite a long list of action points I’m not yet completely satisfied with. But if I put it up, maybe I’ll be able to get some help with it, who knows eh :smiley:


Best regards,
Temaran

Hi everyone so far the only contribution I can made it’s you don’t need to copy the shaders and made a plugin for use it, just change the uproject line “LoadingPhase”: “PostConfigInit”, (was Default), and you can implement **Temara **code directly in you actor class(I don’t remember if you can recompile the classes inside editor).
But I notice that to make this shaders work, the output it’s a texture to use in the materials not a real shader material.
I need to see the source engine code but I don’t really want to =(. I’m using procedural mesh creation successfully and custom shaders with a texture output(Temara) or custom expressions, last it’s the idea of modify MaterialTemplate.usf but this would change all materials behavior. Anyone find a way to use a shader directly on a material instance?

Hello Alex!

The reason for it being a plugin is to make it easy to integrate into any project without having to weave that much, and it makes a good tutorial format :slight_smile:
As you say, it is quite possible to integrate it into a project directly. As for the shaders, I’m afraid that until they integrate my shaders from project level code, or make that functionality available some other way, it is going to be quite hard to compile complex custom shaders any other way than putting them in the Engine/Shaders folder somehow.
As for using the output in engine materials. As you mention, the easiest way to do it is to simply output to a render target like in my tutorial and then reference that rendertarget in your material input.
If you want to create custom HLSL code in your material shaders however, you don’t need my plugin at all! Epic already has a material node that lets you write custom shader code, more info can be found here:

I wasn’t completely able to parse your question, but if I missed answering anything, please let me know :slight_smile:

Best regards,
Temaran

Hi Temaran:

No prob at all =), you did quite nice tut/feature and yes I’m aware of the custom shader code on custom expresion, but I want more control over it XD hahaha. The funny thing : I get some interesting results on editor, the material seems to change to the custom pixel shader output but not in the viewport, seems the material instance on the scene it’s other than the one show in the properties. Anyway Thanks a lot for everything!, Oh just one question did you try FMeshMaterialShader implementation like him:Creating custom shaders and binding them with the material editor - Rendering - Unreal Engine Forums?

Cheers,
Alex

Hello again :slight_smile:

Thank you for the kind words!
I’ve seen the thread you referenced, but I haven’t had use for it yet, but might in the future, who knows!

Best regards,
Temaran

Thank you, very useful!