This bug can’t be fixed by changing that setting, also this setting below supposed to fix it:
Project Settings → Rendering → Uncheck “Render Unbuilt Preview Shadows in game”
But tested it doesn’t have any effect for this.
To clean the watermark it has to either use “Virtual shadow maps (Beta)” instead of “Shadow Maps”, or make all light sources movable and don’t bake any static lighting in UE5.1.
Maybe this is intended to force users to use Lumen and dynamic lighting, because UE5.1 has greatly boosted Lumen performance.