@EvanHammel good eye! Nanite was indeed causing these weird shading artefacts (including the weird reflections in the mirrors/windows!
@comlys Disabling Nanite fixed the issue.
The perceived performance is actually better although the fps is slightly lower. (about 10fps less without Nanite) but a smoother camera. I assume nanite was having issues with the complex nature of the geometry which was then causing performance and shading issues elsewhere.
Bit of a shame, but I suppose I can do some additional optimizations to the model to get back the ~50fps.
Thanks for the hint @EvanHammel !