The final impact depends highly on what you’re doing. What you need to keep in mind is that you render things twice.
Each eye goes through the render pipeline and for example dynamic light will be fully calculated twice making dynamic light in VR quite incredibly expensive.
It’s hard to tell what exactly is causing your performance loss but it is normal to have a significant frame rate drop when running VR.