So, as I understand, you give move task to character, and during task execution, you remove its controller, so task cannot be executed, because there is no controller anymore. What do you expect?
You need some different approach. Maybe AI character, with persistent AIController, which you can give move task by some communication with your player controller.