Ok, now I understand your misunderstanding. I mean smooth as in the smoothing the vertices/normal. Menu->mesh->smooth normals. The polygon count doesn’t change!
Do not however subdivide, unless you wanted to use the subdivide geometry as the base.
No,
Smoothing is referring to the Normal angles rather than the subdivision. Take all the internal edges of your UV group and soften the edges (edges>soften edge i think it is, will double check when i get home) this is akin to making a smoothing group in Max (which is why some refer to it as smoothing). What this does, is for all the soften edges it will average the normal angle, so you will need to rebake both normal and displacement for it. Also the reason I say all internal edges of the UV is that each split uv edge (and edge the is in two different UV islands within your UV setup) is treated automatically as a hard edge.
Can put a video up later if you are having problems still.
Maya and Max are a little bit different in that sense. The “Smooth” operator adds geometry in Maya, and you can specify the amount of divisions. You can “Set Normal Angle…” or “Soften Edge” to your normals, but that didn’t help me alleviate the displacement problem I was having since my normals were at a 180 degree angle to begin with.
As of now my displacement map is correct when I added the Smooth operator to my mesh (like the screenshot above).
I know what the difference is and that’s why it’s explained in Maya format. Adding geo by smoothing your mesh makes it no longer a low poly mesh (will be usually 4x or more the size because it adds 4 quads to every existing quad and smoothing by averaging out the vertices). So having a smoothed mesh isn’t the same as having softened edges correct. Personally I would never make a UV layout like that anyway. Best way to lay them out is to detach the arms (as there is too much of an angle and too close to get a proper cage bake of both normal and displacement) and probably the legs as well.
I just added the Smooth operator to increase the subdivisions for the displacement map, and then deleted it or set it to 0 afterward. I had the same trouble in xNormal too, with my displacement including the edges of the low poly mesh. That software didn’t seem to have a smoothing setting either.
The reason for my UVs to be laid out as one continuous island is to have a minimum amount of seams. The “cuts” are inside the arm and inside the legs so they won’t be noticeable. I’m using Substance Designer to generate procedural elements; blood spatters, dirt, tattoos, etc. and those are randomly placed, so they’ll be more noticeable if I have quite a few UV islands. The head will be on a separate UV tile, so I can get increased resolution for my textures and normal maps. I just did a quick test and threw the islands all on one.
Anyways thanks for your help and suggestions!
The displacement map looks better, as in more smooth looking, but will it map better in the renderer to the base geometry is another question. When you bake normal or displacement, you should bake using the base mesh because, only then will the baking be accurate, & the end result (normal or displacement on base mesh) display correctly.
I have a last suggestion, why not try render out the character using the displacement map you created, & see for yourself if the character render correctly. If it works, then fine.
The problem I had is that I couldn’t bake displacements to give a smooth result. Normal maps looked fine without having to add extra geometry to my low poly mesh. If you have access to Maya and can give me step-by-step instructions I’ll give it a shot. Like I said Im new to displacement mapping so I’m just winging it…