I ended up recreating a map from scratch, making sure nothing has dynamic shadows, merged smalls actors, set all lights to static except the sun directional light, forward rendering, precomputed visibility and it now runs perfectly smooth. I don’t know what caused the issue but I had timings up to 10ms for the visibility pass.
Hope that helps someone in the future !