Normal map not propperly vertexpaint blending

So for the past few days I have been digging deep into vertex painting meshes and make them 5-way vertexpaint ready.
The shader is propperly set up and is 1:1 with the example shown in the official documents. https://docs.unrealengine.com/latest/INT/Engine/UI/LevelEditor/Modes/MeshPaintMode/VertexColor/MaterialSetup/5Way/index.html

In unlit mode everything is fine, but the moment I go into lit mode (may it be with rebuild lighting or dynamic shadows) the normal maps show (random?) seams where the vertex paint should blend them.
Url to example between lit/unlit: http://i.gyazo.com/11c6401c851dfbae60dbee62ff5ac18b.mp4

but on other places its fine:

And the vertex paint is done correctly:

And you can most def. see that the normals aren’t blending:

The diffuse and normal map of the floor are WorldAligned, so the seam doesn’t make any sense.
The UV maps are done correctly, there is no seam in the normal maps texture and the Vertex colors are correct.

I also triple checked the Lightmaps, at 256 I did notice a slight lighting seam, redid the UV to be sure that the floor section + part of the wall is nicely on grid in the 0-1 space but still those normal seams.

Any/all/some/a lot of help is appreciated because I am at a point where I am willing to pull my hair out… and I don’t have any on my head.

Hey Luos -

I believe i know what you issue may be and why it looks fine in some places and not in others. How do your vertices line up along that seam? And by that I mean do the verts line up in a point to point scenario. If they do not you would probably need another detail normal texture overlaying the seam to help hide it. I am looking for a article that was published a while back during the UDK days about how we typically hide mirrored seams, as soon as I find it I will edit this post and link it.

Thank You

Eric Ketchum

The vertices line up exactly along the normal-seam. Thing is, should the vertex paint not blend that?
There is already a detail normal present, ive tested with and without it to see if that was the culprit but it wasnt.

Ive double/tripple/quadruple/hexadruple checked lightmaps, uv maps, material setup and such.

It has something to do with light in combination with the normal (best guess so far)
Also, none of the materials are ever mirrored in this case.

Do you have two vertices from two meshes meeting along that line and not along the line of the place where the blending is correct?

No, everything in view is just one mesh.
If you want I could migrate some stuff to a new folder and send you the material + mesh.

Here is the migrated version:

I had to remove the base textures and normals, but they can easily be replaced in an instance of the material.
Hope that will help :slight_smile:

Hey Luos -

I believe I have tracked down your problem. You have some pretty significant stretching along that curve portion of your wall. (I have attached a screenshot of the lightmap density visualization so you can see what I am talking about). Try to “unstretch” and “unsquash” the Lightmap UV Channel along that seam and see if that doesn’t correct your lighting problem.

Thank You

Eric Ketchum

Sorry picture now …

Hello eric, The stretching comes forth of me testing to see if the seam occurs when the all the vertexes are on grid in the 0-1 space of the lightmap.

My best bet is still that the normals are not propperly blending for some reason.
You can def. see it in the mesh browser when you move around the light.
Here is a video of it:

Both the auto generate UV and my original uv causes this:

As an additional test I expanded the UV map of the floor so it adds a few more polygons on each side moving up onto the wall.
The result is still the same:

This is now the lightmap density:

After this test I again let UE4 build the lightmap for me (tried both create new and layout using channel zero)
And this is the lightmap density:

And lit mode:

And world normal view:

In which you can def. see there is a sharp edge.
while the vertex paint looks like this:

So as I said before, my best bet is still that something messes up when normal maps are blended by vertex paint. (which is also the main reason I passed on the material setup I made to be sure nothing is wrong in that.

Just a small update with some info:
Using a much flatter normal map reduced the seam drastically as can be seen here:

But then again, without normals the whole mesh looks fine.
So that doesn’t take away the fact that for some reason the normals are not blending properly.

Thing is, I am working on this for a market place package.
I cannot permit myself to have those seams around as it would harm reviews.

Tiling the wall along the floor might actually be a decent suggestion which I am going to try asap.
It would mean a few additional vertices on about 150 meshes… but I have the time due to no paypal support atm :slight_smile:

After attempting the extending to floor suggestion Ill let you know how it ends up.

Thanks for all your work Eric!
I cannot express my gratitude enough :slight_smile:

Hey Luos -

Your normal has a tiling issue of a sort. In aligning your floor you are offsetting the normal map such that it is meeting the other normals at a corner and not on a tilable edge. Try to adjust the tiling to allow the tilable edges to remain along the cave walls as close as possible, or blend the wall texture further down along the floor itself.

Ultimately as well if you like the way it looks besides those few seams, you can always place rock props to cover it.

Thank You

Eric Ketchum

Hello erik, I remade the test mesh to allow the wall texture to blend on with the floor texture exactly on floor level (so there is no diagonal blending) but the seam still remains.

Lit mode

Vertex paint mode

Normal view mode

As you can see, the problematic blend now skipped the edge.
which is the new edge to make sure that the blend is on the most flat surface as possible.

Any thoughts?

Hey Luos -

I believe I have discovered the issue and am working up images from my test to show you, but the issue revolves around vertex count. In the area where you have the seam (so along your wall to floor connection) you need to double your vertex count. This will allow smaller and more fine smoothing between the two textures. You don’t need to double your vertex count everywhere just in the seam area. Much like a Character artist would double the edge loops around bending seams for rigging purposes.

Eric Ketchum

Phew :slight_smile:

Yup, that finally did it.
In my case it Means I need to triple the polycount in regions where the vertex color needs to blend properly.
Which also means I have to practially redo all cave meshes I did so far.
Oh well. *puts on his sigh here we go again cap.

Biiig hug
thanks!

Hey Erik, After some more testing the past few days I did notice that adding more polygons where the seam is does hide it decently, but the seams where still there.

Today me and Jan Kaluza (Kazhaar) sat down and see what the problem could be.
After dissecting the whole 5way material blend we noticed that in the example (https://docs.unrealengine.com/latest/INT/Engine/UI/LevelEditor/Modes/MeshPaintMode/VertexColor/MaterialSetup/5Way/index.html ) there was no good indication (or at least not easily noticed) in the image that “white” is not R,G and B to 255 but should be black + alpha 100 instead of white + alpha 100.

This in my case also caused some issues with blending two normals as well as one of the nodes I used to set the world position for the normals… which was not derived from a “world aligned normal” but a regular world align, which doesn’t have some of the math needed to properly blend the two normals.

After solving these two issues (The vertex color and the world aligned normal) there is actually no need for more polygons.

Hello Erik, Even though I am going to make a new post about this…
since its somewhat related I am also adding this link here:

It shows an issue between multiple UV channels, vertex paint and normals resulting in seams.