Thank you @nande. I went with the second approach as well. I instantiate Manager dynamically from GameMode and safe the reference in it. The chance of it being added again or deleted is then minimal. As you suggested all goes through GameMode.
Previous suggestion with WorldSubsystems is also interesting. Looks very nice and definitely something to look into in the future.