Help with VR grenade issue, only working on world X vector

PLEASE help me solve this problem. this single issue is preventing me from releasing this to the marketplace.
I am trying to pull a grenade pin from the grenade with the secondary hand, And I have got it so close to what I want it to work like, But its only working correctly when the grenade is facing a POSITIVE X world location.
I want the pin to be restricted like I have made it to a single Vector for removal But not care what world rotation it has. I have tried to fix this but without luck.
here is a video and a BP pic of the key area