I figured it out and feel stupid for it, for some reason when I playtest with the Oculus Plugin enabled it sees the engine itself as a VR application which isn’t the case, and tries to render the preview window over the engine or something.
In the Oculus VR menu I have to select add window select one of the 2 Unreal Engine Windows and if I select the correct one it doesn’t lag anymore.