Interestingly it seems to work now. I have UV-unwrapped every object once more and played a bit around with the “margin” setting within blender to increase the space between UV shells and now it seems to be working. I tried that before, but I guess something went wrong last time.
I can send you the UE4 file if you want to have a look at it. My lighting still looks weird, if you have some time on your hands and nothing else to do, you can have a look at my model.
Is there a way I can DM you the file? As the building I have modeled is a replica of my high school, I could get into trouble for publishing it online which I want to avoid ^^