Controller inputs in VR stop working randomly within Editor

I have a problem with VR controls in UE 5.3 where the controllers get tracked but randomly stop taking input (there is no pattern as far as I can see), it happens in random points of the project and maybe one of 10 times. It is not bound to specific controllers as I have tested with different models and it still happens.