I was able to Track it down to the raytracing in combination with the Lumen Global Illumination.
I think unreal tried to raytrace something on every surface in my gameworld (open world)
After I disabled the hardware raytracing the in the Project Settings, everthing seams working like normal again.