Increasing Focal Length on Camera Killing FPS

When I increase the focal length the framerate comes to a crawl and I don’t think it’s the dof effect because it does it with high f-stops and little dof present.

I am using adaptive tessellation so maybe it is something to do with the engine tessellating everrthing without regard to the new frustrum achieved with a narrower field of view?

Is there a way to get my frames back without a wide angle all of the time?