How to improve lightmass shadow quality?

That looks like screenspace ambient occlusion. You can use a Post Process Volume to tweak that.