What are the correct settings for baking and exporting normal maps to unreal engine 4? So far i couldn’t get proper results, tried every option in ue4 and most options in 3d coat, results were bad. Tried to flip green channel, no luck.
Most of the normal map should be pretty close to the 128, 128, 255 purple color. The light green and dark purple gradients you see on the texture means it’s treating the face like it’s facing 90 degrees the wrong way. This could be a baking issue or a vertex normal issue, but it’s hard to say for sure. Some gradients, like you in the bottom left with the larger square with the 2 squares, is normal and expected, but extreme gradients are a sign there’s something wrong.
Unfortunately 3DCoat is one of the few applications I use regularly, but I haven’t every baked in. The preferences look correct.
If it looks good in 3d coat, then i assume that synced workflow could produce same results in engine. I should also mention that retopo mesh is not completely flat on its sides, it produces gradients on normal map.
The problem with having such intense normals in your texture is that you are then relying on the normal map transformation to be able to handle too much and the gradients have to be absolutely correct or problems show up. It can magnify what were only small discrepancies between what the expected and actual normals. SFJohn and Zac definitely have the right idea that your normals shouldn’t be trying to use the normals to handle a mesh with so much difference between the smoothed normal and actual surface orientation. Generally artists add what are called ‘supporting edge chamfers’ to make the flat parts actually flat in both models as much as possible.
Since I haven’t used 3dcoat the only thing I can suggest is trying the same mesh with another available package like blender or 3dsmax if you really want to fix this mesh as-is.