'Update Instance Transform' / 'Remove Instance' == big performance hit even if used only on 1 instance

It almost seems like if I remove an instance on one it still redraws every instance in the scene.