Going from non VR mode to VR mode having bad performance

Yes, exactly that. I have a normal map load first, if they decide they can go into the VR map by pressing a key. The VR map has bad performance once loaded, dropping under 45fps, but works totally fine if the VR map is the first map loaded.