It seems like the hitch has been fixed in 4.10 if you possess with a controller instead of unpossessing. So I am able to switch them back and forth with the AI retaking the other pawn when I want to switch back.
But it would be nice to be able to remove the AI controller all together as its still not needed.