Scene laggy until I finish looking around

Some more info. It is a small interior scene, low poly meshes and low resolution textures (max 1024)
It has static lighting. It runs on Android, ASTC, GLES 3.1 on a high end phone.

With your comment do you mean to check my lightmap UVs? How can I troubleshoot this?