I have this case where I would need the fresnel effect for a sort of rim lighting, where fresnel would work as if the normals of the object are somewhat different. To better explain what I mean I have simplified the case to the following:
Sphere on the left has no smoothing groups, sphere on the right has one smoothing group. This results in their normals being different.
Top image is what I get currently and bottom image (which is photoshop mocap) is what I would like to get for the sphere without smoothing groups, which is the fresnel rim light being “rendered” or applied to the object like if it has the normals of the sphere on the right. But for all other shading purposes, the sphere without smoothing groups should be rendered correctly like it should if it has no smoothing groups.
I tried many different ideas and I understand it’s very illogical thing to want from the renderer and it very well may be technically impossible.
I would be happy for whatever crazy workaround you could come up with, ideally without modifying the source code. It could be done in mesh material or even in post processing.
Thanks
edit: even confirming that this probably isn’t possible would help me a lot
I don’t think that’s going to work, ultimately because of the angles.
But, what you could do if you wanted to get that result is to create a copy of the mesh–so you’d have a green one without smoothing and then another one that’s slightly bigger with smoothing and then on the second one you could set it up so it’s transparent except for the fresnel effect, then I think it would appear correctly–though it would have Z-fighting issues if you move a certain distance away because it’s so close to the mesh.
There might be an alternative if you know how to use the material editor well enough where maybe you could use a smoothed normal map to modify the fresnel effect without changing the shading. I wouldn’t know how to do that though or if that’s even possible.
Thanks for the input.
Yeah, the thing with copy of the mesh didn’t work out too well, including what you mentioned about z fighting.
I tried to do something with a smoothed normal map, fresnel nodes (there are two different) even have an input for normal map, but that also didn’t work too well.
I’m afraid this is probably technically impossible (without some nasty hacks), but would help to get confirmed that this is a no go.
Hi,
this idea will only work for a sphere (or other simple geometries that you know).
You should be able to sample the world position and subtract the objects/components/actors world position from it. This gives you a vector pointing outwards like your smoothed normal would. This vector can be normalized so you can apply the fresnel like before.
The idea with the smoothed normal map actually sounds better (since it’s a general approach). Could you explain what you did and why it didn’t work so we could try to get this to work together.
Hope this helps 
I’m wondering if you could use normal mapping INSTEAD of smooth grouping…
It’s possible to bake normal maps to non-smoothed objects. Not often recommended, but still. Maybe you could recreate the smoothing by taking your smooth-grouped object as the “high poly” and baking its normals to the flat one? This would make the flat one appear smoothed, but then you could simply not feed the normal map to the fresnel and it would generate it like it was the simple mesh…
EDIT: wait, sorry, that would give you the opposite effect, the smoothed sphere with the faceted fresnel
Why not use the smooth mesh but render it flat shaded in-engine?
This keeps the vertex count low and avoides the extra effort of baking a normal map.
I didn’t know about flat shading being possible via those few nodes. I’m not sure this completely solves my problem, but maybe you can help me out a bit more.
The thing is that at the end I need to shade some faces as if they have one smoothing group and other faces as if they have another smoothing group - e.g. left half of the sphere = smoothing group 1, right half of the sphere = smoothing group 2. Would that be somehow possible by modifying the shader you posted above? Because otherwise it would be good solution.
edit: and thanks for the previous ideas too
Well, you would need some kind of a mask (like a texture or vertex colors) to blend between the flat shaded and smooth shaded parts.
In this case, baking a normal map is the better solution I guess.
That being said can you post a pic of your actual mesh? Is it just a regular model with some hard edges or do you really need whole parts of the model to be flat shaded?
Thanks , some good ideas.
The object could be something like this (screenshot from 3dsmax):
With this kind of mesh I’d definitely go for baking a normal map.
Since baking to a mesh with lots of hard edges/smoothing groups requires separated UV shells -which will shrink the usable UV space- you could also keep the mesh fully smooth shaded and bake down the hard edges. Then use the normal map regularily in the UE4 material but calculate the fresnel effect based on vertex normals.
Yeah, I will try to mix up a few ideas, including those provided in this thread.
Thanks again.
