Normal Map Workflow with Xnormal is not working

Using the Unreal 4 Documentation found here [HERE][1], I am unable to bake a good normal map using the documentation’s settings. I have to change the settings quite drastically to achieve a bake that works, and even then, the bake is not quite perfect, with some surface shading errors still occuring. My workflow:
High / Low / Cage / and Triangulate in 3Dsmax 2013 and 2014(tried both), 1 smoothing group for the mesh. Reset xforms on meshes.
Export triangulated .fbx 2013 files to xnormal with same exact settings as documentation, baked using exported normals and cage.
Import same triangulated .fbx into UE4, broken normals appear in the model previewer and in viewport.

I should note, the hard line that appears at the edge facing the camera, is present in the high poly. That is working as intended in the bake. The shading errors with the red X’s are the problem areas.

  • My files: [FILES][2]

https://dl.dropboxusercontent.com/u/2344272/crit/UE4_NormalMap_Error.jpg

Hi Quack,

I am attempting to re-produce your issue here and have been unable thus far to do so. Would you mind walking me through the exact steps you took so I can better assist you? Since you have already provided your assets I just need your import steps to the Unreal Editor. Do you have this issue on any other meshes that you have imported or that are currently in the Editor? Thank you and have a great day!

Adam

Hey Adam,

Thanks for the response!

I created 2 cubes on two different computers from scratch, high poly with beveled edges, low poly. I used every setting exactly as your documentation says, except for when you import into unreal. The import dialog prompt in your documentation is different then in Unreal4. You can not choose explicit normals any more, you only have a choice to calculate tangents, import normals only, or import normals and tangents. I am assuming that ‘import normals’ means explicit normals?

Regardless, the only way to get an ok bake is to not follow your documentation. Your documentation states to uncheck the “Tangents and Binormals” FBX checkbox when exporting to xnormal, then to unreal. UN-Checking this box breaks the normal map bake. This image is from another polycount member with the same issue ( http://www.polycount.com/forum/attachment.php?attachmentid=16371&d=1395448501 )

To get a good bake, I have to check the fbx box for Tangents and Binormals and then make sure on import to use "Import Normals and Tangents. Otherwise I get those triangle errors.

I have tried this on two different computers with 3 different meshes and gotten the same issues. I can get the normals to look good, just not following the recommended workflow.

Hi Quack,

After discussing this issue with colleagues, we discovered that these documents may be describing a workflow that no longer applies. It is being taken into consideration by the developers. Thank you for pointing this out!

Have a great day!

Adam

Ok cool. I just wanted to make sure the results I am getting with exporting with tangents and binormals and going through xnormal is the optimal path.

Again, thanks for the help.

I just corrected a discrepancy with how UE4 uses tangents to how xNormal does. This further corrects things. A better document explaining the correct way of generating normal maps for UE4 should be coming.

Will this fix be in UE4.1 ? I’m also having the same problems as Quack, but I’ll wait for the fix to try baking from Modo to Xnormal to UE4.

I was about to post about this. Here is a test I did: Epic’s Xnormal documentation vs a workflow I have in Maya

Here is a document I wrote of my maya workflow incase anyone might find it useful.

I’m still having the same issue mention by Quack in UE4.2. I tried going from Modo to Xnormal to UE4, and Maya 2015 to Xnormal to UE4 - both instance I get that nasty triangle artifact. So I started looking at the difference between 2 similar files both with 1 smoothing group, one where UE4 generates the tangents & binormals and the other with imported tangents & binormal, there was a huge difference on 2 of the vertex(though I only highlight 1 spot in the pic), might this be what’s causing the triangle bug?:

http://s4.postimg.org/w8b8jeld8/Tangents.jpg

Here are my results, as you can see I get a triangle on 2 of the corners:

So my workflow for the red cube is:

  1. Maya 2015: Export all mesh(low,high,cage)using FBX 2014 with 1 smoothing group(with export smoothing group ON and Tangents & Binormals OFF) for the cage and low res mesh, all transform freeze for all meshes and only the cage and low res mesh are triangulated
  2. Baking is done in Xnormal 3.18.8 with Mikk tspace with “Compute binormal in the pixel shader” checked. Low res = use exported normals, High res= used exported normals,use external cage file provided in the zip. Bake to 16 bit TIFF at 2048 X 2048 pixel. Normal map Swizzle Coordinates = X+ Y- Z+.
  3. Take the output normal map TIFF file into Photoshop, change it to 8bits/channel(Image>Mode>8Bits/Channel),save file as PNG no compression.
  4. Import the same low res mesh into UE4 with the “import normal” option.
  5. Import PNG normal map into UE4
  6. Create a basic Material with color, metalic,roughness and apply the nomal map to the normal slot.

Here are the files: http://www.firedrive.com/file/1D4671885E09284B

Hi SonKim -

Please try running again but do not triangulate your cage. See if the same error occurs and let us know. Using your assets I was also able to eliminate this issue by breaking up the Cube unwrap to add padding between each island.

We are continuing to look into this.

Eric Ketchum

Hi Eric Ketchum,

Thanks for the answer! Secondly, the low res and cage has to be the same when using Xnormal(so I we would need to triangulate the cage, since the low res is triangulated). I think the confusion comes from the following line:

“One is to give your model a single smoothing group. This will create a very gradient heavy normal map but few seams on UV edges.”

The latter part “few seams on the UV edges” make its sound like you can get away(in my case) with 1 UV island for my cube. That’s appears to be a incorrect assumption.

I guess the Normal Map Creation Guide, needs to add some additional information like, if you are using 1 smoothing group but have sharp corners(90 degree turns) on your mesh, you’ll need to split the UVs at those edges.

This isn’t ideal because it creates more UV vertices, compare to using a single UV island.

This should be mention in the guide also another option for those with access to 3DSmax/Maya you can export “Smoothing Groups” and “Tangents and Binormals”, you’ll be able to use 1 smoothing group and 1 UV island, yet still get perfect normal map shading. Baking done in Xnormal.

Another option on the horizon is handplane3D(www.handplane3d.com) which should let you use 1 smoothing group, lay out the UVs how you see fit(1 or multiple UV island) by just using “Import Normals” in UE4. UE4 support is coming to handplane3d.

My cube(1 smoothing group,multiple UV island, Import Normals):

Quack’s “cube”(1 smoothing group,multiple UV island, Import Normals):

My cube(1 smoothing group,1 UV island, Import Normals and Tangents):

http://s1.postimg.org/wuyw1r1ny/Normal_not_sync.jpg

I’ve been struggling to get this working today, but I’ve found a solution that got the desired results. If you have a low poly mesh consisting of just six faces (a cube) where the high poly mesh has some nice rounded bevels, you need to isolate each face in the low poly cube’s UVs to get a nice bevel at the right place.

When I unfolded the UV of the cube into one single UV island and one smoothing group, I got the exact same artifacts with a diagonal line in the shading. When I split the faces into separate islands, the artifacts disappeared.

The normal map renderer (xNormal, Maya, 3dsmax etc) uses the smoothing group layout of the low poly mesh in its calculation process. If you have one smoothing group for your cube, you’re essentially telling the normal map renderer that your cube is round. The renderer will then create a normal map that tries to even out the roundness of your low poly cube to match the geometry of your high poly mesh. But with a normal map of limited bit depth (and compression on top of that), I guess it’s very hard for it to match the smoothing group shading perfectly; hence the artifacts.

The fix you describe is not the solution to this problem.

The problem you ran into is standard normal map creating steps.

The problem I encountered is a different problem, highlight by my image.

You gave good advice, but it doesn’t apply to this issue. I am posting this, not to say you are wrong, but to make sure that people who come here with my problem are focused towards the answer.

It’s been many months since I encountered this issue and will take the time tonight to see if it has been fixed and post a resolution.

Here’s the skinny on things. There’s a particular edge case with UE4’s tangent and bitangent calculation code that creates a problem where the tangents are split when you choose to import normals and calculate tangents for your static mesh. Other MikkTSpace implementations (Blender, Xnormal, Toolbag 2) don’t have the same issue, and if you bake a map in Xnormal it will show up fine in either Blender or Toolbag 2 (as long as you have “Calculate binormal in the pixel shader” checked.) See the image below.

If you import the .fbx into Blender, then export the .fbx from Blender with its calculated tangents and bitangents with the v7 binary exporter, Unreal will import the correct tangent space rather than calculating the incorrect one, and will display the mesh with correctly mapped normals.

I’m working on stepping through the code to see if I can figure out the problem, but I think you guys at Epic should be able to find it faster than I can. I do believe that it’s a problem with your FBX importer’s tangent calculation, and it’s probably a really silly edge case. I’ve also put a brief summary in the official FBX problem reporting thread. I think we can fix this bug once and for all.

I’m having similar issues with FBX files from Modo. Exporting with vertex normals and no tangents/binormals (Modo doesn’t support exporting tangents/binormals anyway), then telling UE4 to calculate the tangents, results in several tangent splits where there should be none.

You can see the very simple cube mesh here in Modo, with UV boundaries highlighted (top). And the resulting mesh in UE4 (bottom) has several extraneous tangent splits (circled in blue) causing very ugly seams across the mesh (highlighted in dotted purple).

Normal map baked in xNormal using a cage and “compute binormals in pixel shader” enabled. All other programs I load this into that support MikkTSpace work perfectly fine.

In my own tests, it looks like the 90-degree angles have more to do with breaking the tangents than the UV splits.

I’m not sure. I’ve tried some tests based on that and I can’t seem to get 90 degree angles to do anything strange. I get tons of tangent splits where I have UV splits, though. Give this mesh a look over, should show what I mean.

Bottom row is just a bunch of planes that meet at different angles. Next row up is those joined up to triangles at the side. Next row up is the same, but unwrapped with UV splits and BAM tons of tangent splits. Top one is them as closed surfaces and there’s tangent splits with those too because of the UV splits.

Angle Test Mesh

Hello Quack and Everyone Else -

I have been trying to get a really good reproduction of this particular reported issue. From my investigation, this appears to be functioning exactly as intended in UE4 and in all the modeling programs. Before everyone roars in outrage, let me explain what I am seeing and how I got there so we can talk from the same space.

To begin, I think everyone understands how the vertex normal is derived, but to clarify the vertex normal is the unit vector that is perpendicular to the surface (for our discussion this is an ok definition, though vertex normals are slightly different than typical surface normals). From my investigation I cannot see where the vertex normal either derived or calculated comes into the engine as incorrect (I should note here that I am using 4.5.1 for my tests). If anyone is having Vertex Normals computing or derived (imported) coming into the engine as incorrect please post pictures of the vertex normal in your modeling program and the vertex normal in the static mesh editor and, if possible, upload the fbx of your mesh.

The problem that people are having, it seems from reading responses here and over on Polycount, is that the tangents are seeming to import correctly and are calculated incorrectly. In either case the math behind this is very simple to checkout, the tangent of the vertex should run along the + U unwrap starting from the vertex, or more technically, it is the vector which shows increasing U space along each surface. This is very important to understand as no matter what we do to our unwraps the UV space is always perpendicular to each other and always runs 0 to 1.

When you are calculating the tangents, the engine does not look at your smoothing information from Blender, Modo, Max or Maya because you are telling it not to look at it. All that it has to go off of is the unwrap and the vertex itself. Often times this will cause the engine to treat the vertex as multiple vertices because of how you have unwrapped that mesh. Ultimately though the engine looks at the vertex in teh context of your unwrap and assigns as many normals and tangents as it needs.

As a case study of this I used a Pyramid, like JedtheKrampus does over at Polycount. I am going to be showing you Maya images but I confirmed this behavior in both Max(2013) and Blender(2.72). To begin here is my mesh, it shows you hard edges:

And here is the same mesh completely Smoothed in maya:

Now the unwrap for the mesh:

So the smoothed mesh brought into UE4 with imported Normals and Tangents looks like this:

As you can see the Normals and tangents are identical between Maya and the engine. Now lets look at the Calculated Normals:

Looking at the unwrap shown with the Calculated normals and you can see the tangents is based on vertex and unwraps planes connected to the vertex that are not merely triangulated quads, so the bottom plane counts in calculation terms as one plane.

So from the examples I have seen, the tangents appear correct when calculated based on the various unwraps and meshes.

If you feel I’ve missed something or if you have evidence that the tangents are not calculating as I describe above, please let us know with screenshots and uploading an FBX.

Thank You

Eric Ketchum


Regarding the initial question from Quack - I would ask as I did earlier but I believe I did not address you directly and I apologize for that, did you triangulate your mesh before running normals? Often time triangulating your mesh before running a normals calculation will result in a normal where the triangulation is running an opposing direction which will cause the original triangulation to show up as a render error. But we have worked on and continue to work on updating the documentation so if you feel some workflow we are using is incorrect, please let us know.

Hmm, I’m not convinced. I actually fixed the issue in my local fork of the engine code.

It’s also telling that this only happens with static meshes and that the skeletal mesh calculates tangents correctly for each vertex.

Edit: Also, you are saying that the engine goes off just the unwrap and the vertex, but looking through the code, it goes off the vertex position, normal, UVs and the faces smoothing groups. (And if it wasn’t using this information, then it would be doing it wrong.)

If you check my last image towards the end of this thread (currently 3rd post from the bottom) you’ll see it’s blending the tangents for the cube correctly;
http://www.polycount.com/forum/showthread.php?t=141659

And I have an FBX here that clearly and reproducably shows the issue;

I’ll try and post up a version from my copy of the engine soon to show what it should look like. Or you could just rig it and import it as a skeletal mesh.

I do triangulate before all of my bakes. But I may have missed it. Because of the fact that this is still open, I will devote a bit of time tonight to try with the latest version of Unreal.

I recently imported my District9 mech with which I stress tested edges to see how far I could push it, and the normal map seemed almost on par as it did in Toolbag2.

I will report back.