Announcement

Collapse
No announcement yet.

Extending Custom HLSL / Custom Expressions

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    Extending Custom HLSL / Custom Expressions

    Until we see some more love for fast, custom HLSL in Unreal, here is a discussion of my recently discovered tricks to go beyond the current limits.

    #1 - Custom Functions

    Modifying the Engine's USF files to define custom functions triggers an overhaul shader recompilation of thousands of materials. This is unusable for creative iteration. Also #include-ing files outsides the Engine's Shaders directory crashes the Editor at startup. CustomExpression nodes wrap your code inside CustomExpression#() functions, and that normally prohibits defining your own functions.

    However, there seems to be a barely documented feature in HLSL that allows defining functions (methods) inside struct definitions. struct definitions can be nested inside functions (like the wrapper CustomExpression#).

    So in your CustomExpression Code you can do:

    Code:
    struct Functions
    {
    
      float3 OrangeBright(float3 c)
      {
          return c * float3(1, .7, 0);
      }
    
      float3 Out()
      {
        return OrangeBright(InColor);
      }
    
    };
    
    Functions f;
    return f.Out();
    Create and connect an input pin "InColor" of float3 on the CustomExpression node. Any Node inputs passed into CustomExpression#(), like InColor above is available inside nested function definitions.

    The cool part is, this is all happening inside your own effective namespace, not interfering with Unreal's USF files, and the compilation is Fast for iteration. So now, you can start to build a library of custom functions, and more complex shaders. It seems HLSL is prohibiting defining a struct nested inside a struct, so make sure to define your custom structs above and outside struct Functions.


    #2 - External Code Editing and #include

    Instead of editing intricate code and custom libraries inside the little primitive textbox of CustomExpression, you can edit them in a better external editor with syntax highlighting, code navigation etc, and #include that file. So if you put the above code in a file named Test.hlsl, you can:

    Code:
    #include "Your Path...\Test.hlsl"
    return 0;
    // enter spaces here and press enter to retrigger compilation
    The dummy "return 0;" is to tell CustomExpression node that this not a single line expression but a full function body. The spaces will be required to signal the CustomExpression textbox that it changed, and pressing enter will compile your externally changed and saved Test.hlsl. Of course, you can split the external file and the dynamic code portion, if you prefer to make quick changes and compiles inside the textbox.


    #3 - Multiple Outputs

    The CustomExpression node limits the output to a float4. I have some ideas using CustomExpression#() chains and injecting macros to redefine their definitions and calls later to create additional outputs or complete Material outputs later at CalcPixelMaterialInputs() etc. But let's see how the above tricks work for people, before we delve into that.


    #4 - Multiple Passes

    Some image processes like convolution are far more efficient when broken into multiple passes by storing intermediate results in a texture. Currently, if you have a process that comes before convolution, a previous function has to be called redundantly for all the neighborhood kernel samples. There is some hope in Unreal's render targets, but whether they can render multiple passes per frame synced, etc, and how much more spaghetti they will create with additional pass materials and control blueprints, we should discuss... Ideally, texture writing should be available completely inside the Material Editor, there should be variables, true branching, and custom #includes at the global scope, all features requested for years to take the already excellent default PBR system in Unreal to the next level with industry leading oceans, volumetric clouds, professional chroma key, etc.

    #2
    Branching inside editor and more outputs from custom node had been requested for quite some time indeed. Though personally, I would prefer a convenient code shader authoring system side by side with material editor.
    Last edited by Deathrey; 03-29-2017, 07:44 PM.

    Comment


      #3
      these are some great findings. the Multiple Outputs is something I had been needing recently so it'd be great if you could shed some light on it
      Help me back! follow me on Twitter
      Developer of Elium - Prison Escape

      Comment


        #4
        I've given MultiOut more thought this morning, and ran some tests. I have a design that would be quite easy to use with one #include for your custom program, another #include with the preprocessor magic. You would set the path of where you keep your includes in the first CustomExpression node, and chain additional CustomExpression nodes for additional float outputs. Your program will run first, outputting an array of floats of a given count, and the additional CustomExpression nodes in the chain will output individual floats from that output array. Only limitation in this easy-to-use design is that you would only have one MultiOut program per material.

        Comment


          #5
          Great finds Büke! I was wondering, what if I'd like to use a relative path to the external .hlsl file? Do we know which folder it would be relative to?
          Zoltan Erdokovy, Sr. Technical Artist, Aimotive

          Comment


            #6
            Got it: It's the 'Engine\Shaders' folder.
            Zoltan Erdokovy, Sr. Technical Artist, Aimotive

            Comment


              #7
              Hello, Buke could you please give a more detailed example as how you realize MultiOut? I still don't quite understand. Thanks a lot.

              Comment

              Working...
              X