That can have multiple causes:
- Your Material has the Lighting Model set to Subsurface or similar, causing the backlight to shine through
- the Light/Sun could be reflected. turn the roughness up and the specular down to disable reflections.
- Do you have a SkyLight placed in the Map? If so, make sure the lower hemisphere is set to black.