Hello to everyone!
Well, I’m writing this post because I need some help regarding to vertex shaders and .usf files.
First of all, what I’m trying to achieve is an outline around my meshes. I know that I can do that with a postprocess material, but the result I’m looking to achieve is different.
I want to get an outline with the “old-school” style. This style I’m talking about is the one you can get by making a slightly bigger copy of your mesh and flip normals, so you’ll
get a nice outline effect around your mesh. And because of the different angles between the mesh vertices and different mesh shapes, you can observe distinct thickness for
diverse views of the mesh, that’s what I’m looking to achieve.
As it’s natural, the first thing I did was making a bigger copy of my mesh and flip normals, so I got exactly what I wanted.
But this have two main problems:
(For the rest, I’ll tell my progress in doing that, and the issues I have. If you actually know a way of doing it without duplicate meshes and want to share it, you can skip
the following text and post it, I’ll really appreciate it :))
-
Performance: In my project I have relatively big levels with lots of items and physics attached to them (both static and skinned meshes), and making a copy of each
one significally lowers fps because, literally, I have double number of meshes in my level, which implies duplicate physics, double number of vertices to calculate, double
number of particle physics, etc. So, I end up with a very poor performance. -
Rigging issues: When it comes to attach a skeleton to a mesh, in principle I can copy the same bone weight information to the outline copy of the mesh, so it can follow the
movement pretty well, but in certain points I want to make the outline change thickness. Is then when I can get some weird behaviours, not very noticeable, but sometimes
it’s very annoying. That, not mentioning that is time consuming in order to fix the bone weights.
After this, I tried to make my own material layering attributes, one material with bigger size using vertex normal and the other with a texture and a toon shader.
I couldn’t find a way to flip normals inside the material editor, neither to mix at the same time two materials (I don’t want blending materials, I want superposed). I tried here using
also custom nodes with HLSL, but without success.
So, the only way I can think of (and it should be the easiest) is writing my own vertex shader. It should be simply draw the mesh with a toon shader, and
then, the same mesh but bigger size and flipped normals.
I did a lot of research and finally I found a way of writing my own shaders changing some source files of the engine, mainly in the folder of shaders
with .usf files. And now I have my own toon shader, that works perfectly. I did it defining my shader in the file “ShadingModels.usf” and changing others in
order to make it recognizable by the engine. The problem is that it’s a pixel shader and not a vertex one. I haven’t found a way of changing the vertex positions writing HSLS code.
Then my question is, can someone give me, at least, a hint of how to write a vertex shader for Unreal Engine in wich I can change the vertex positions and normals?
I’ve made my research and the information regarding to write shaders in UE is almost non-existent. So I truly think this will help a lot of people.
I would really appreciate the help 
Thank you very much for your time and consideration