I noticed the same thing in one of my projects. It only occurs when I use a specific character mesh. I am not sure what causes it, but surely there has to be another solution that does not involve turning of Virtual Shadow Maps?
There has to be something specific about the character mesh that can be rectified instead.