Normal map not working with 2 different UV-Channels


im currently trying to create modular wall pieces that fit together seamlessly with no visible borders. After having a lot of issues creating my UVs in the right way i ended up jus splitting my static mesh on two different UVs. One is used for the front side and takes up all of the 0-1 space to guarantee it fits together with other instances of my modules. The other channel is used for all the other parts.;base64

Im am basicly using 2 different versions of the same material with a TextureCoordinat-Node to apply it to my 2 used channels:;base64

This actually worked surprisingly well and i got my modules fit together perfectly. However for some reason the materials normal map only seems to work for 1 of the 2 used channels. Currently i only get it to work on my sides and the front segment seems to just have the base color applied. Its two identical copies of the same material just sent to two different channels.

Does anyone know why the normal map isnt working on both channels? Is there maybe some sort of limitation that prevents you from using different normal maps on different channels? Also i know the way im doing the whole thing with having the same material twice probably isnt how its supposed to be done. Is there a better way to just apply 1 material to both of my UV-Channels?
I really would like to stick to the two different UV-Sets because that currently makes the creation of my modules a lot easier.