Shadow Depth is taking more than 50% of the scene for Archviz

Please correct If I’m wrong, but isn’t the Shadow-Depth map only used for movable objects? I have a scene that is completely static (besides 2-3 little boxes) and the ShadowDepth is consuming most of the computation time in my scene. How do I best go about this? Btw. I’m using the lightstage BP from the ‘realistic Rendering’ example.