I personally solved this by using a different light channel for the directional (sun) light. That way Im able to control what mesh is affected by the external light and skylight. For areas that shouldn’t have any sun light e.g. a cave, this works like a charm.