What can I do to make the hand hold the grip of the gun even while the character is bending downward or upward?

You will need to show the skeleton hierarchy and the node linkages for forward/up/down (from the animation preview view) to be able to debug this.
Chances are, the parenting of the objects is going wrong somewhere, either in linkage, or in how the data is interpreted.