Can't get player controller if the VR spectator is not in the level.

So whenever I delete the VR spectator blueprint from the level (inside the VR template)
I can not control the VR pawn anymore after it spawns in the level.

I don’t understand, what am I missing here?
Is there a reason I can’t control the player (VR pawn) without it?