Yea well i mean it looks good without shadows but 
i have 600m elevation differences on this 8x8km map so having no shadows would be kinda “strange” wouldnt it?
Btw before someone says its a blocky hightmap: 16 bit raw and i also tried to make 32 bit raw reduce it in PS back to 16 in hope it would get smoother, but it seems its really the shadow casting with lumen