Prevent CharacterMovement CurrentFloor from colliding with VR Hands

I am implementing the UltraLeap hand tracking into a character instead of a pawn for multiplayer and it looks like the CharacterMovement → CurrentFloor calculation is sometimes being calculated based on hit result with my VR Hands. Is there a way to prevent objects from being used in that calculation?

When I call CharacterMovement → CurrentFloor → BreakFindfloorResult → HitResult → HitComponent → GetDisplayName it shows the floor was calculated based on a hit with my VRHands.

I set the hands to not collidable but didnt seem to make a difference.