If they are invisible and with the collision off i think it wont be as bad because he doesnt process it gor most of the calculations, although replication might still be a problem because it still needs to update it. Unless you disable replication while its invisible and reenable when its visible.