English is not my first language so I will try my best to describe it

So basically, after I apply the heightmap to material, the shadow of my character is way below than my character (basically, the body and shadow don't connect at the feet, the shadow is lower)
I tried to change height ratio but even the smallest value (0.055, if I go lower it might crash) it is still a problem

Is there anyway to fix it? What caused it?
My theory is that the heightmap I'm using is actually too "white" (not sure tho, I purchased the texture which includes its heightmap), is there anyway to darken the heightmap in general?