Vertex Shaders

Hello to everyone!

Well, I’m writing this post because I need some help regarding to vertex shaders and .usf files.

First of all, what I’m trying to achieve is an outline around my meshes. I know that I can do that with a postprocess material, but the result I’m looking to achieve is different.
I want to get an outline with the “old-school” style. This style I’m talking about is the one you can get by making a slightly bigger copy of your mesh and flip normals, so you’ll
get a nice outline effect around your mesh. And because of the different angles between the mesh vertices and different mesh shapes, you can observe distinct thickness for
diverse views of the mesh, that’s what I’m looking to achieve.

As it’s natural, the first thing I did was making a bigger copy of my mesh and flip normals, so I got exactly what I wanted.
But this have two main problems:

(For the rest, I’ll tell my progress in doing that, and the issues I have. If you actually know a way of doing it without duplicate meshes and want to share it, you can skip
the following text and post it, I’ll really appreciate it :))

  • Performance: In my project I have relatively big levels with lots of items and physics attached to them (both static and skinned meshes), and making a copy of each
    one significally lowers fps because, literally, I have double number of meshes in my level, which implies duplicate physics, double number of vertices to calculate, double
    number of particle physics, etc. So, I end up with a very poor performance.

  • Rigging issues: When it comes to attach a skeleton to a mesh, in principle I can copy the same bone weight information to the outline copy of the mesh, so it can follow the
    movement pretty well, but in certain points I want to make the outline change thickness. Is then when I can get some weird behaviours, not very noticeable, but sometimes
    it’s very annoying. That, not mentioning that is time consuming in order to fix the bone weights.

After this, I tried to make my own material layering attributes, one material with bigger size using vertex normal and the other with a texture and a toon shader.
I couldn’t find a way to flip normals inside the material editor, neither to mix at the same time two materials (I don’t want blending materials, I want superposed). I tried here using
also custom nodes with HLSL, but without success.

So, the only way I can think of (and it should be the easiest) is writing my own vertex shader. It should be simply draw the mesh with a toon shader, and
then, the same mesh but bigger size and flipped normals.

I did a lot of research and finally I found a way of writing my own shaders changing some source files of the engine, mainly in the folder of shaders
with .usf files. And now I have my own toon shader, that works perfectly. I did it defining my shader in the file “ShadingModels.usf” and changing others in
order to make it recognizable by the engine. The problem is that it’s a pixel shader and not a vertex one. I haven’t found a way of changing the vertex positions writing HSLS code.

Then my question is, can someone give me, at least, a hint of how to write a vertex shader for Unreal Engine in wich I can change the vertex positions and normals?

I’ve made my research and the information regarding to write shaders in UE is almost non-existent. So I truly think this will help a lot of people.

I would really appreciate the help :slight_smile:

Thank you very much for your time and consideration :slight_smile:

Hey, you said you realized your nono, is it a custom shading model? I would like to ask how to customize the model can do?

Take a look at the vertex factory shaders.

Flipping normals method isn’t actually flipping the normals but vertex winding.(which in side effect give you mirrored normals). This is same as swapping cull mode from back face to front face.

The method you describe is how I made the outline shader we used in Gears, but it was mostly only used for a few items at a time, such as the placeable traps and barriers in horde mode.

A few notes:
a) you don’t need a unique mesh created. You can just use a copy of the same mesh and set the material to be 2-sided. Then you simply use the VertexNormal * ScalarParam to decide the thickness. Then you use something like depth fade to control the opacity. Using varying numbers here can lead to interesting halographic looks.
b) only spawn duplicates for objects in your scene that are currently highlighted. It would be very perf wasteful to do it everywhere.
c) As you guessed, this method is not nearly as fast as the post process method since it it renders more polygons and adds a bit of translucent overdraw.
d) Hard edged normals will have issues since you will see the faces separate. The only way around that I know of is to use soft vertex normals.