Trees are very bright if Visible in Ray Tracing is turned off

How about converting your trees to nanite and use virtual shadow maps.

In Unreal fest they showed about better support for nanite foliage and shadows and stuff, I’m using 5.5 at the moment so can’t confirm any improvement by this method.

Since i see you are working in 5.6 maybe try doing something that Unreal showcased in the fest.