Help with Maya displacement

I have a high-res model and a UV’d low-res character mesh. Excuse the UV layout since I did a quick test of it. When I use Transfer Maps in Maya my displacement comes out like below. I think I need to use subdivisions or a cage, but I’m relatively new to rending out maps in Maya. Any help would be appreciated. Thanks!


Have you seen this video?


Have you seen this video?

I did see that, and I used the same steps which gave me the result above.

I get the same problem in xNormal too.

It looks right to me, though there’s probably errors on some things like the fingers and other parts where surfaces are really close to each other.

Well Im trying to take a high res mesh and create a displacement from that, based on the UVs on the low res mesh. Is that not possible? I thought the workflow was similar to how to create normal maps.

Right, is the UV layout not what you set it?
The weirdness of the displacement map is because after it subdivides the mesh it has to push out the parts where it added more geometry.
I think though it looks like it would only work on a mesh without smoothing, so you might need to adjust the baking settings so that the low-resolution mesh has smoothing applied and accounted for when you generate the displacement map.

The UV’s are correct. I thought I would get a smoothed gray-scale image. It seems that it’s showing my edges of my low-res model.

So I need to increase the geometry of my low res mesh in order to get smooth displacement from the high res one?

It looks like I needed to sub-divide my low-res mesh to get it smooth.

Hi Adam,

I undeleted this post so that it can be helpful for anyone else that may have the issue.

Thank you!


Sure thing.

For those of you who might stumble upon the same problem, what you need to do is apply a ‘Smooth’ operator to your low-res mesh. Here is the before and after with a “Division Level: 2”. I hope that helps someone!

I was going to ask if you apply smoothing. If you want to make normal map, you usually have to smooth the UVs (in Maya). Guess its same with displacement.

Anway, why are you using displacement map & not normal map?

The only time I smoothed my low poly mesh was when I was creating the displacement. When it came to normals I didn’t need to do that, I just chose the high-res mesh as the target and created the normal map.

Well I never used a displacement map for a character, so I wanted to try it out. The displacement didn’t work as I initially thought. It “ballooned” my character when I applied tessellation, so I’m going to stick with normal maps. It will probably be effective for wrinkles and such, but for what I outputted above it didn’t matter since my low-res topology matched the high-res surface.

I am going to use distanced-based tessellation for my characters with PN triangles to smooth out my mesh when the camera is close to it. So with that and normal maps that should be enough.

You shouldn’t subdivide the mesh to generate the displacement–that would mean that the displacement would only look correct on the subdivided mesh.
What it needs is that the mesh has smoothing applied to where it doesn’t look faceted. I don’t know about the texture baking settings, but in some baking tools there’s a setting that will use the smoothed mesh for baking.

There was no option for that in the “Transfer Maps” rollout in Maya, so sub-diving the low-res mesh prior to creating the map seemed to do the job like my screenshots showed. If there’s another/better way to do it in Maya please let me know. I’m new to displacement, and I couldn’t find anything out on the internet regarding it.

I don’t use Maya, in 3ds Max it will use whatever smoothing is applied to the mesh and in Mudbox it has an option to generate the maps with smoothing applied.

For Maya, you 100% need to smooth verts on low poly mesh. I am a Maya user (but migrating soon), & I made tonnes of normal maps in Maya, so definately true that you need to smooth your verts, otherwise you get wierd result.

The telltale signs are ‘darken’ edges.

Also you cannot have already a normal map apply, as it will take it also into consideration when generating normal map. I am refering to the Transfer map features of Maya. If you use Turtle or something else, I am not sure.

You should NOT subdivide, you should smooth both high & low res version of the meshes you wanted to transfer between.

Also, try adjust the CAGE until you get just enough to cover the entire mesh. A Cage that is too wide or too narrow can have issues when transferring Normal and displacement. If you need to push & pull verts in the Cage, so so. Most of the time, adjusting the value will do however.

Isn’t smoothing the same thing as subdividing a mesh? You’re adding extra geometry. In any case the Smooth operator works with a division level in Maya. Thanks everyone!