I tried Adjusting the shadow resolution and that had no effect and I tried enabling the shadow caching and that had very little effect.
How can I ever get this to run better?
Edit: I have been running the stat ShadowRendering and depending on which direction I am facing, It can go up and can go down. Facing the direction where there is Movable lights means I am at 75-80 ms while facing away means I am down at 8ms so it is definitely the point lights.