I just thought to put it out here so people that have the same issue as me don’t have to search for 2 days.
I had a couple of problems that were already known on the forums and answer pages (share privilege settings on the network) but I still had the issue where I could connect to every slave but they didn’t build any lighting. After putting the log settings on verbose I noticed that my slaves where constantly lossing the connection with my PC. I tested multiple things and I finally got it to work when I just outright disabled the windows firewall on my PC.
So my problem laid with a firewall setting.
On the bottom of this UE4 article it states that swarm especially needs ports 8008 and 8009. After adding those rules in the firewall for both ports allowing tcp connection, bob’s your uncle it works.
Hopes this helped someone, or maybe future me when I forget this.