Delay in between letting go and falling in VR Climbing System with Momentum

Hello,
I am working on a climbing system in VR that has momentum when the player pushes off from the wall. The problem I am having is that there is a delay once the player lets go and falls in the direction they push off in. The code is below for the falling vector calculation and the point where the delay occurs.
Does anyone know how to fix this?