Normal map not propperly vertexpaint blending

So for the past few days I have been digging deep into vertex painting meshes and make them 5-way vertexpaint ready. The shader is propperly set up and is 1:1 with the example shown in the official documents.

In unlit mode everything is fine, but the moment I go into lit mode (may it be with rebuild lighting or dynamic shadows) the normal maps show (random?) seams where the vertex paint should blend them. Url to example between lit/unlit: [video][/video]

but on other places its fine: [video][/video]
And the vertex paint is done correctly: [video][/video]
And you can most def. see that the normals aren’t blending: [video][/video]

The diffuse and normal map of the floor are WorldAligned, so the seam doesn’t make any sense. The UV maps are done correctly, there is no seam in the normal maps texture and the Vertex colors are correct.
I also triple checked the Lightmaps, at 256 I did notice a slight lighting seam, redid the UV to be sure that the floor section + part of the wall is nicely on grid in the 0-1 space but still those normal seams.
Double checked the normals as well, none are inverted/flipped/messed up afaict.
Any/all/some/a lot of help is appreciated because I am at a point where I am willing to pull my hair out… and I don’t have any on my head.

answerhub link: Normal map not propperly vertexpaint blending - Platform & Builds - Unreal Engine Forums

Its been a few days, and even though I got a reply from Erik on answerhub, still no solution to the problem.
In the meantime I have been doing other things but slowly getting to a point where my problem really needs to be solved.