The note 4 is capable of handling quite a few polygons, if you keep the draw calls low.
Try also removing the skybox and replacing it with an unlit sphere, if you haven’t yet.
Have a look to the post process volume in the vr template for an hint to which effects should be removed.