Ladder volume glitchy if hit collision while climbing..?

I’ve recently implemented climbable ladders in my game (using LaddVolume). They work great, except if i hit geometry while climbing up the ladder. It seems to instantly push the pawn back down (sometimes enough that it kills the pawn from falling damage, even though the distance is very small).

Anyone else had these issues?