So I have actually managed to narrow this down to the Owen mesh itself. It only seems to cause poor performance when using that mesh. I thought maybe it had something to do with the cloth simulation but disabling it made no difference…So I have no idea what it is other then it has something to do with the Owen mesh.